issue_comments

Table actions

6 rows where author_association = "MEMBER" and issue = 1172229856 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 3

issue 1

  • Should the zarr backend support NCZarr conventions? · 6 ✖

author_association 1

  • MEMBER · 6 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1090499559 https://github.com/pydata/xarray/issues/6374#issuecomment-1090499559 https://api.github.com/repos/pydata/xarray/issues/6374 IC_kwDOAMm_X85A_7Pn shoyer 1217238 2022-04-06T17:04:26Z 2022-04-06T17:04:26Z MEMBER

As it is currently it is also not possible to write a zarr which follows the GDAL ZARR driver conventions. Writing the _CRS attribute also results in a TypeError:

Can you elaborate? What API are you using to do the write: python, netcdf-c, or what?

This error message comes from Xarray and can be triggered by calling to_zarr(): https://github.com/pydata/xarray/blob/facafac359c39c3e940391a3829869b4a3df5d70/xarray/backends/api.py#L162

I don't think netCDF-C needs to be involved at all, which is why I suggested opening a separate issue.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Should the zarr backend support NCZarr conventions? 1172229856
1090464275 https://github.com/pydata/xarray/issues/6374#issuecomment-1090464275 https://api.github.com/repos/pydata/xarray/issues/6374 IC_kwDOAMm_X85A_yoT shoyer 1217238 2022-04-06T16:25:40Z 2022-04-06T16:25:40Z MEMBER

@wankoelias could you kindly open a new issue for writing GDAL ZARR?

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Should the zarr backend support NCZarr conventions? 1172229856
1076810559 https://github.com/pydata/xarray/issues/6374#issuecomment-1076810559 https://api.github.com/repos/pydata/xarray/issues/6374 IC_kwDOAMm_X85ALtM_ rabernat 1197350 2022-03-23T20:54:39Z 2022-03-23T20:54:39Z MEMBER

Sure, to be clear, my hesitancy is mostly just around being reluctant to maintain more complexity in our zarr interface. If there is momentum to implement and maintain this compatibility, I am definitely not opposed. 🚀 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Should the zarr backend support NCZarr conventions? 1172229856
1076796582 https://github.com/pydata/xarray/issues/6374#issuecomment-1076796582 https://api.github.com/repos/pydata/xarray/issues/6374 IC_kwDOAMm_X85ALpym shoyer 1217238 2022-03-23T20:38:12Z 2022-03-23T20:38:12Z MEMBER

@DennisHeimbigner I think it would be great to standardize NCZarr as a super-set of the "Xarray-Zarr" standard! I think Xarray should indeed be able to read such files. If you want to read a sub-group, you can read the sub-group in a separate call to xarray.open_zarr().

@rabernat I would not be opposed to adding support inside Xarray for reading NCZarr data, specifically to understand NCZarr's encoding of dimension names when using Zarr-Python. This wouldn't give 100% compatibility with NCZarr, but it would be very close (maybe just with incorrect dtypes for attributes) with a minimal amount of work. I don't think it would be a big deal to look for .nczvar files.

 
{
    "total_count": 3,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 2,
    "rocket": 0,
    "eyes": 0
}
   Should the zarr backend support NCZarr conventions? 1172229856
1076622767 https://github.com/pydata/xarray/issues/6374#issuecomment-1076622767 https://api.github.com/repos/pydata/xarray/issues/6374 IC_kwDOAMm_X85AK_Wv rabernat 1197350 2022-03-23T17:39:57Z 2022-03-23T17:39:57Z MEMBER

My opinion is that we should not try to support the nczarr conventions directly. Xarray already supports nczarr via netCDF4. If netCDF4 can open the Zarr store, then Xarray can read it.

Supporting nczarr directly would require lots of custom logic within xarray. That's because nczarr introduces several additional metadata files that are not part of the zarr spec. These additional metadata files break the abstractions through which xarray interacts with zarr; working around this requires going under the hood, access the store object directly (rather than the zarr groups and arrays).

I would turn this question around and ask: if netCDF4 supports access to these datasets directly, what's the advantage of xarray bypassing netCDF4 and opening them directly? If there are significant performance benefits, I would be more likely to consider it worthwhile.

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Should the zarr backend support NCZarr conventions? 1172229856
1076592711 https://github.com/pydata/xarray/issues/6374#issuecomment-1076592711 https://api.github.com/repos/pydata/xarray/issues/6374 IC_kwDOAMm_X85AK4BH max-sixty 5635139 2022-03-23T17:14:30Z 2022-03-23T17:14:30Z MEMBER

CC @pydata/xarray 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Should the zarr backend support NCZarr conventions? 1172229856

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);