home / github / issue_comments

Menu
  • GraphQL API
  • Search all tables

issue_comments: 786837666

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/issues/2799#issuecomment-786837666 https://api.github.com/repos/pydata/xarray/issues/2799 786837666 MDEyOklzc3VlQ29tbWVudDc4NjgzNzY2Ng== 17484729 2021-02-26T19:06:08Z 2021-02-26T19:07:20Z NONE

Thanks all for your prompt responses!

@hmaarrfk , I share your recommendation and it's a great thing to be able to fallback to numpy arrays when the algorithmic part is well decoupled from the data preparation process. It's what I also do when I can. However, in workflows working on streaming data the two things (data preparation and computation) may be intricated or frequently alternated. My example of "buffer data array" structure is something quite natural to consider in such a context and having an efficient implementation of labelled ndarray could really serve the task.

@shoyer I think a first "lite" implementation fully implemented in python could be already a great thing. It would not achieve numpy performance, but the additional cost du to management of coordinates alignement should not be too expensive.

An additional suggestion: if the target is computational workflows, trying to have some compatibility with packages such as eagerpy would enabling working with other tensor frameworks commonly used in machine learning. This kind of feature could be adressed yet in another package, but having it in mind may influence the early choices in term of implementation (ex: pure python vs C++).

@jhamman, @shoyer I would be pleased to share my work on buffer data array if you think it could serve as kind of use-case. In this context, I experimented a bit with a « crafted » lite version of xarray and I could achieve a x10 factor in performance improvement.

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  416962458
Powered by Datasette · Queries took 0.727ms · About: xarray-datasette