issue_comments

Table actions

3 rows where issue = 233350060 and user = 1197350 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

  • rabernat · 3 ✖

issue 1

  • If a NetCDF file is chunked on disk, open it with compatible dask chunks · 3 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
632294837 https://github.com/pydata/xarray/issues/1440#issuecomment-632294837 https://api.github.com/repos/pydata/xarray/issues/1440 MDEyOklzc3VlQ29tbWVudDYzMjI5NDgzNw== rabernat 1197350 2020-05-21T19:19:50Z 2020-05-21T19:19:50Z MEMBER

It seems to me that the there are lots of "layers" of "chunking", especially when you are talking about chunking an entire dataset,

To simplify a little bit, here we are only talking about reading a single store, i.e. one netcdf file or one zarr group. Also out of scope is the underlying storage medium (e.g. block size). 

{
    "total_count": 2,
    "+1": 2,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   If a NetCDF file is chunked on disk, open it with compatible dask chunks 233350060
632266536 https://github.com/pydata/xarray/issues/1440#issuecomment-632266536 https://api.github.com/repos/pydata/xarray/issues/1440 MDEyOklzc3VlQ29tbWVudDYzMjI2NjUzNg== rabernat 1197350 2020-05-21T18:23:13Z 2020-05-21T18:23:13Z MEMBER

Can we overload the chunks argument in open_xxx to do this? We are already adding support for chunks="auto" ...

This gets tricky, because we may want slightly different behavior depending on whether the underlying array store is chunked.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   If a NetCDF file is chunked on disk, open it with compatible dask chunks 233350060
632183683 https://github.com/pydata/xarray/issues/1440#issuecomment-632183683 https://api.github.com/repos/pydata/xarray/issues/1440 MDEyOklzc3VlQ29tbWVudDYzMjE4MzY4Mw== rabernat 1197350 2020-05-21T16:13:46Z 2020-05-21T16:14:08Z MEMBER

We discussed this issue today in our pangeo coffee break. We think the following plan would be good:

  • [ ] Write a function called auto_chunk(variable) which examines a variable for the presence of a chunks attribute in encoding or within the data itself. Returns a new variable with chunked data.
  • [ ] Refactor open_zarr to call this function
  • [ ] Add it also to open_dataset to enable auto-chunking of netCDF and geotiff data

Should we have an option like chunk_size='native', or chunk_size='100MB', with chunks chosen to align with source chunks.

 
{
    "total_count": 2,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 2,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   If a NetCDF file is chunked on disk, open it with compatible dask chunks 233350060

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);