issue_comments

Table actions

6 rows where author_association = "CONTRIBUTOR" and issue = 1368740629 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 1

issue 1

  • Generalize handling of chunked array types · 6 ✖

author_association 1

  • CONTRIBUTOR · 6 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1554332081 https://github.com/pydata/xarray/pull/7019#issuecomment-1554332081 https://api.github.com/repos/pydata/xarray/issues/7019 IC_kwDOAMm_X85cpTmx tomwhite 85085 2023-05-19T10:01:06Z 2023-05-19T10:01:06Z CONTRIBUTOR

Thanks for all your hard work on this @TomNicholas!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Generalize handling of chunked array types 1368740629
1477628301 https://github.com/pydata/xarray/pull/7019#issuecomment-1477628301 https://api.github.com/repos/pydata/xarray/issues/7019 IC_kwDOAMm_X85YEtGN tomwhite 85085 2023-03-21T10:54:51Z 2023-03-21T10:54:51Z CONTRIBUTOR

I would like to get to the point where you can use xarray with a chunked array without ever importing dask. I think this PR gets very close, but that would be tricky to test because cubed depends on dask (so I can't just run the test suite without dask in the environment

I just released Cubed 0.6.0 which doesn't have a dependency on Dask, so this should be possible now.

 
{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 1,
    "rocket": 0,
    "eyes": 0
}
   Generalize handling of chunked array types 1368740629
1469907455 https://github.com/pydata/xarray/pull/7019#issuecomment-1469907455 https://api.github.com/repos/pydata/xarray/issues/7019 IC_kwDOAMm_X85XnQH_ tomwhite 85085 2023-03-15T12:21:23Z 2023-03-15T12:21:23Z CONTRIBUTOR

I think it might make sense for me to remove the CubedManager class from this PR and instead put that & cubed+xarray tests into another repo. That keeps xarray's changes minimal, doesn't require putting cubed in any xarray CI envs, and hopefully allows us to merge the ChunkManager changes here earlier.

That sounds like a good plan to me.

Places dask is still explicitly imported in xarray

There are a few remaining places where I haven't generalised to remove specific import dask calls either because it won't be imported at runtime unless you ask for it, cubed doesn't implement the equivalent function, that function isn't in the array API standard, or because I'm not sure if the dask concept used generalises to other parallel frameworks.

  • [ ] open_mfdataset(..., parallel=True) - there is no cubed.delayed to wrap the open_dataset calls in,
  • [ ] Dataset.__dask_graph__ and all the other similar dask magic methods
  • [ ] dask_array_ops.rolling - uses functions from dask.array.overlap,
  • [ ] dask_array_ops.least_squares - uses dask.array.apply_along_axis and dask.array.linalg.lstsq,
  • [ ] dask_array_ops.push - uses dask.array.reductions.cumreduction

This is a useful list! I hope that we could close the gap for some of these over time.

I would like to get to the point where you can use xarray with a chunked array without ever importing dask. I think this PR gets very close, but that would be tricky to test because cubed depends on dask (so I can't just run the test suite without dask in the environment)

Agreed. I have opened https://github.com/tomwhite/cubed/issues/154 to make it possible to test without a Dask dependency.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Generalize handling of chunked array types 1368740629
1463469875 https://github.com/pydata/xarray/pull/7019#issuecomment-1463469875 https://api.github.com/repos/pydata/xarray/issues/7019 IC_kwDOAMm_X85XOscz tomwhite 85085 2023-03-10T08:50:10Z 2023-03-10T08:50:10Z CONTRIBUTOR

Great work @TomNicholas!

I don't have a strong opinion about the tests, but putting them in a new project to keep xarray changes to a minimum is probably a good idea for the moment.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Generalize handling of chunked array types 1368740629
1285349000 https://github.com/pydata/xarray/pull/7019#issuecomment-1285349000 https://api.github.com/repos/pydata/xarray/issues/7019 IC_kwDOAMm_X85MnN6I tomwhite 85085 2022-10-20T11:15:50Z 2022-10-20T11:15:50Z CONTRIBUTOR

@TomNicholas it might be good to rebase this now that #7067 is in.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Generalize handling of chunked array types 1368740629
1255158880 https://github.com/pydata/xarray/pull/7019#issuecomment-1255158880 https://api.github.com/repos/pydata/xarray/issues/7019 IC_kwDOAMm_X85K0DRg tomwhite 85085 2022-09-22T15:03:34Z 2022-09-22T15:03:34Z CONTRIBUTOR

I think the manager keyword will also need adding to open_zarr, open_dataset and to_zarr.

I'm interested in trying this out on some of our genomics use cases in sgkit (see https://github.com/pystatgen/sgkit/issues/908), so please let me know when you think it's ready to try @TomNicholas.

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Generalize handling of chunked array types 1368740629

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);