issue_comments

Table actions

2 rows where author_association = "MEMBER" and issue = 614814400 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 2

issue 1

  • Document Xarray zarr encoding conventions · 2 ✖

author_association 1

  • MEMBER · 2 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
632924467 https://github.com/pydata/xarray/pull/4047#issuecomment-632924467 https://api.github.com/repos/pydata/xarray/issues/4047 MDEyOklzc3VlQ29tbWVudDYzMjkyNDQ2Nw== rabernat 1197350 2020-05-22T21:58:19Z 2020-05-22T21:59:09Z MEMBER

Thanks for the useful questions @DennisHeimbigner

Suppose I am given an array X with shape(10,20,30) and an _ARRAY_DIMENSION attribute on X with the contents _ARRAY_DIMENSION=["time", "lon", "lat"] Then this is equivalent to the following partial netcdf CDL: netcdf ... { dims: time=10; lon=20; lat=30; ...} Correct?

Yes, correct

I assume that if there are conflicts where two variables end up assigning different sIzes to the same named dimension, then that generates an error.

Yes, correct as well. Understanding how this works requires me to describe some xarray internals. When decoding a Dataset, each array is decoded as an xarray.Variable. According to those docs "a single Variable object is not fully described outside the context of its parent Dataset". The Zarr decoding process returns a Variable, which is basically a tuple of dims, data, attributes, encoding, where dims is the list we got from _ARRAY_DIMENSIONS.

Once the variables have all been decoded, then we put them together into a Dataset object. At that point, if there are inconsistent shapes across the different variables, an error will be raised. So far we haven't encountered this situation, because all the Zarr data we read tends to have been also written by Xarray, so it is consistent. But you could definitely manually hack a Zarr store to break this consistency, rendering it un-decodable by Xarray.

Finally it is unclear where xarray puts these dimensions. In the closest enclosIng Group? or in the root group?

I hoped this was clear in the documentation I wrote which is now live here: http://xarray.pydata.org/en/latest/internals.html#zarr-encoding-specification. What I said was

To accomplish this, Xarray developers decided to define a special Zarr array attribute: _ARRAY_DIMENSIONS. The value of this attribute is a list of dimension names (strings), for example ["time", "lon", "lat"]. When writing data to Zarr, Xarray sets this attribute on all variables based on the variable dimensions. When reading a Zarr group, Xarray looks for this attribute on all arrays, raising an error if it can’t be found. The attribute is used to define the variable dimension names and then removed from the attributes dictionary returned to the user.

An "array attribute" has a specific meaning in Zarr: it is the user metadata associated with an individual array. So the _ARRAY_DIMENSIONS attribute lives in the .zattrs file of each Zarr array. It is not a group-level attribute.

As you pointed out on the last call, there are clearly some downsides to having chosen to store this important property with the rest of user metadata (.zattrs). However, it allowed us to move forward without any changes to the zarr spec.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Document Xarray zarr encoding conventions 614814400
631603771 https://github.com/pydata/xarray/pull/4047#issuecomment-631603771 https://api.github.com/repos/pydata/xarray/issues/4047 MDEyOklzc3VlQ29tbWVudDYzMTYwMzc3MQ== dcherian 2448579 2020-05-20T17:03:57Z 2020-05-20T17:03:57Z MEMBER

Thanks @rabernat 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Document Xarray zarr encoding conventions 614814400

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);