issue_comments

Table actions

5 rows where author_association = "MEMBER" and issue = 868352536 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 3

issue 1

  • Zarr encoding attributes persist after slicing data, raising error on `to_zarr` · 5 ✖

author_association 1

  • MEMBER · 5 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
839106491 https://github.com/pydata/xarray/issues/5219#issuecomment-839106491 https://api.github.com/repos/pydata/xarray/issues/5219 MDEyOklzc3VlQ29tbWVudDgzOTEwNjQ5MQ== rabernat 1197350 2021-05-11T20:08:27Z 2021-05-11T20:08:27Z MEMBER

Instead we could require explicitly supplying chunks vis the encoding parameter in the to_zarr() call.

This could also break existing workflows though. For example, pangeo-forge is using the encoding.chunks attribute to specify target dataset chunks.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Zarr encoding attributes persist after slicing data, raising error on `to_zarr` 868352536
839090347 https://github.com/pydata/xarray/issues/5219#issuecomment-839090347 https://api.github.com/repos/pydata/xarray/issues/5219 MDEyOklzc3VlQ29tbWVudDgzOTA5MDM0Nw== shoyer 1217238 2021-05-11T20:02:58Z 2021-05-11T20:02:58Z MEMBER

It occurs to me that another possible fix would be to ignore chunks from pre-existing encoding attributes in to_zarr(). Instead we could require explicitly supplying chunks vis the encoding parameter in the to_zarr() call.

Probably better to remove encoding more aggressively overall, but this might be a good intermediate step...

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Zarr encoding attributes persist after slicing data, raising error on `to_zarr` 868352536
830974512 https://github.com/pydata/xarray/issues/5219#issuecomment-830974512 https://api.github.com/repos/pydata/xarray/issues/5219 MDEyOklzc3VlQ29tbWVudDgzMDk3NDUxMg== shoyer 1217238 2021-05-03T00:52:21Z 2021-05-03T00:52:21Z MEMBER

Somewhat inevitably, I finally fit this issue this week, too :)

I am increasingly coming to the conclusion that there are no "safe" manipulations in xarray that should preserve encoding. Perhaps we should just drop encoding after every operation in Xarray.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Zarr encoding attributes persist after slicing data, raising error on `to_zarr` 868352536
828071017 https://github.com/pydata/xarray/issues/5219#issuecomment-828071017 https://api.github.com/repos/pydata/xarray/issues/5219 MDEyOklzc3VlQ29tbWVudDgyODA3MTAxNw== rabernat 1197350 2021-04-28T01:26:34Z 2021-04-28T01:26:34Z MEMBER

we probably would NOT want to use safe_chunks=False, correct?

correct

The problem in this issue is that the dataset is carrying around its original chunks in .encoding and then xarray tries to use these values to set the chunk encoding on the second write op. The solution is to manually delete the chunk encoding from all your data variables. Something like python for var in ds: del ds[var].encoding['chunks']

Originally part of #5056 was a change that would have xarray automatically do this deletion after some operations (such as calling .chunk()); however, we could not reach a consensus on the best way to implement that change. Your example is interesting because it is a slightly different scenario -- calling sel() instead of chunk() -- but the root cause appears to be the same: encoding['chunks'] is being kept around too conservatively.

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Zarr encoding attributes persist after slicing data, raising error on `to_zarr` 868352536
827990207 https://github.com/pydata/xarray/issues/5219#issuecomment-827990207 https://api.github.com/repos/pydata/xarray/issues/5219 MDEyOklzc3VlQ29tbWVudDgyNzk5MDIwNw== mathause 10194086 2021-04-27T22:36:42Z 2021-04-27T22:36:42Z MEMBER

Thanks for the clear error report. On master you should be able to do ds2.to_zarr("test2.zarr", consolidated=True, mode="w", safe_chunks=False) - see #5065

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Zarr encoding attributes persist after slicing data, raising error on `to_zarr` 868352536

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);