issue_comments

Table actions

2 rows where author_association = "MEMBER" and issue = 887711474 sorted by updated_at descending

✖
✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

issue 1

  • Inconclusive error messages using to_zarr with regions · 2 ✖

author_association 1

  • MEMBER · 2 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
845604377 https://github.com/pydata/xarray/issues/5290#issuecomment-845604377 https://api.github.com/repos/pydata/xarray/issues/5290 MDEyOklzc3VlQ29tbWVudDg0NTYwNDM3Nw== shoyer 1217238 2021-05-21T02:22:15Z 2021-05-21T02:22:15Z MEMBER

The (potentially very large) coordinate still needs to fit in memory though... either when creating the dummy zarr store (which could be done differently) or when opening it. Is that correct?

This is correct

That wont work for my use case when the coordinate is very large. Do you know an alternative? Would it help if I store the coordinate with a non-dimension name?

Yes, this would work.

We do probably want to change this behavior in the future as part of the changes related to https://github.com/pydata/xarray/issues/1603, e.g., to support out of core indexing. See also https://github.com/pydata/xarray/blob/master/design_notes/flexible_indexes_notes.md

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Inconclusive error messages using to_zarr with regions 887711474
839069111 https://github.com/pydata/xarray/issues/5290#issuecomment-839069111 https://api.github.com/repos/pydata/xarray/issues/5290 MDEyOklzc3VlQ29tbWVudDgzOTA2OTExMQ== shoyer 1217238 2021-05-11T19:45:01Z 2021-05-11T19:45:37Z MEMBER

Hi @niowniow thanks for the feedback and code example here. I've been refactoring the Zarr region re-write functionality in https://github.com/pydata/xarray/pull/5252 so your feedback is timely.

It might be worth trying the code in that PR to see if it changes anything, but in the best case I suspect it would just give you a different error message.

To clarify: - Writing to a region requires that all the desired variables already exist in the Zarr store (e.g., via to_zarr() with compute=False). The main point of writing to a region is that it should be safe to do in parallel, which is not the case for creating new variables. So I think encountering errors in this case is expected, although I agree these error messages are not very informative! - "chunks in coords are not taken into account while saving!?" is correct, but this is actually more a limitation of Xarray's current data model. Coords with the same name as their dimension are converted into an in-memory pandas.Index object -- they can't be stored as dask arrays. You actually still write the data in chunks, but you have to do it by supplying the encoding parameter to to_zarr, e.g., ds.to_zarr(path, mode='w', encoding={'x': {'chunks': 10}}, compute=False, consolidated=True).

If you have any specific suggestions for where the docs might be clarified those would certainly be appreciated!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Inconclusive error messages using to_zarr with regions 887711474

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);