issue_comments

Table actions

7 rows where author_association = "NONE" and issue = 484592018 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 1

issue 1

  • to_zarr append with gcsmap does not work properly · 7 ✖

author_association 1

  • NONE · 7 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
525015784 https://github.com/pydata/xarray/issues/3251#issuecomment-525015784 https://api.github.com/repos/pydata/xarray/issues/3251 MDEyOklzc3VlQ29tbWVudDUyNTAxNTc4NA== jminsk-cc 48155582 2019-08-26T20:20:05Z 2019-08-26T20:20:05Z NONE

Closing since this seems to be a zarr/ zarr consolidation issue and not an xarray issue. 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   to_zarr append with gcsmap does not work properly  484592018
525008289 https://github.com/pydata/xarray/issues/3251#issuecomment-525008289 https://api.github.com/repos/pydata/xarray/issues/3251 MDEyOklzc3VlQ29tbWVudDUyNTAwODI4OQ== jminsk-cc 48155582 2019-08-26T19:59:32Z 2019-08-26T20:02:00Z NONE

Upon further testing it looks like the error has to do with the consolidated zarr store. Even thou the orginal set was written with:

python ds.to_zarr(store=gcsmap, mode='w-', consolidated=True)

The new append to zarr with:

python ds.to_zarr(store=gcsmap, mode='a', consolidated=True, compute=False, append_dim='time')

Fails since it reads an empty array. This might be more of a problem for zarr itself and not xarray. Going to test rewriting a netcdf file to zarr with consolidation and if the error persists then I will move to zarr for the error. 

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   to_zarr append with gcsmap does not work properly  484592018
524912344 https://github.com/pydata/xarray/issues/3251#issuecomment-524912344 https://api.github.com/repos/pydata/xarray/issues/3251 MDEyOklzc3VlQ29tbWVudDUyNDkxMjM0NA== jminsk-cc 48155582 2019-08-26T15:42:58Z 2019-08-26T15:42:58Z NONE

If that is the case then would this also break for s3fs and azures bucket fs package (never used azure not sure about this statement)? Since they all should return a mutable map that gets turned into a string. 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   to_zarr append with gcsmap does not work properly  484592018
524903609 https://github.com/pydata/xarray/issues/3251#issuecomment-524903609 https://api.github.com/repos/pydata/xarray/issues/3251 MDEyOklzc3VlQ29tbWVudDUyNDkwMzYwOQ== jminsk-cc 48155582 2019-08-26T15:20:53Z 2019-08-26T15:20:53Z NONE

Went back to gcsfs 0.0.4 and the current code still does not work. Going back more requires major changes to environment that conda cannot handle and would not be compatible with the current build of xarray. I think changing mutable maps to strings in this case has never worked with gcsfs and more than likely has not been seriously used until the append to zarr was added. 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   to_zarr append with gcsmap does not work properly  484592018
524892591 https://github.com/pydata/xarray/issues/3251#issuecomment-524892591 https://api.github.com/repos/pydata/xarray/issues/3251 MDEyOklzc3VlQ29tbWVudDUyNDg5MjU5MQ== jminsk-cc 48155582 2019-08-26T14:52:54Z 2019-08-26T14:52:54Z NONE

Even with the changes in gcsfs syntax the same error still occurs since the old syntax just uses get_mapper() under the hood. I can continue to work on finding which version of gcsfs works with the current xarray code but the pinning on gcsfs, xarray, and dask on conda/conda-forge ends up getting out of sync very quickly and will take me a bit more time then I thought it would. 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   to_zarr append with gcsmap does not work properly  484592018
524868941 https://github.com/pydata/xarray/issues/3251#issuecomment-524868941 https://api.github.com/repos/pydata/xarray/issues/3251 MDEyOklzc3VlQ29tbWVudDUyNDg2ODk0MQ== jminsk-cc 48155582 2019-08-26T13:52:08Z 2019-08-26T13:52:08Z NONE

I will look through previous versions of GCSFS to see where the change would have occurred (I need to anyways to get my code working). I will update this comment when I find the version that wants the mutable map to be a string and not keep it as a mutable map. 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   to_zarr append with gcsmap does not work properly  484592018
524367631 https://github.com/pydata/xarray/issues/3251#issuecomment-524367631 https://api.github.com/repos/pydata/xarray/issues/3251 MDEyOklzc3VlQ29tbWVudDUyNDM2NzYzMQ== jminsk-cc 48155582 2019-08-23T15:54:04Z 2019-08-23T15:54:04Z NONE

Yes my bad copying scratch code from an ipython terminal I missed the assignment. It should be

python gcsmap = mapping.GCSMap(gcs_bucket_name) I reedited above to correct that issue.

The version of gcsfs is 0.3.0

conda list out:

gcsfs 0.3.0 py_0 conda-forg

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   to_zarr append with gcsmap does not work properly  484592018

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);