issue_comments

Table actions

6 rows where author_association = "MEMBER" and issue = 745801652 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 2

issue 1

  • Serialization issue with distributed, h5netcdf, and fsspec (ImplicitToExplicitIndexingAdapter) · 6 ✖

author_association 1

  • MEMBER · 6 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
730467523 https://github.com/pydata/xarray/issues/4591#issuecomment-730467523 https://api.github.com/repos/pydata/xarray/issues/4591 MDEyOklzc3VlQ29tbWVudDczMDQ2NzUyMw== rabernat 1197350 2020-11-19T15:54:38Z 2020-11-19T15:54:38Z MEMBER

This is fixed by intake/filesystem_spec#477.

However, the existence of this issue points to the need for more ecosystem-wide integration testing of xarray / dask / zarr / fsspec. I know we discussed this is on some other issue, but I can't find it...

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Serialization issue with distributed, h5netcdf, and fsspec (ImplicitToExplicitIndexingAdapter) 745801652
730031761 https://github.com/pydata/xarray/issues/4591#issuecomment-730031761 https://api.github.com/repos/pydata/xarray/issues/4591 MDEyOklzc3VlQ29tbWVudDczMDAzMTc2MQ== shoyer 1217238 2020-11-18T23:56:17Z 2020-11-18T23:56:17Z MEMBER

OK, I think I understand what's going on. Xarray serializes arguments that should suffice to recreate/open a backend-specific file object (e.g., h5netcdf.File). So if you pass in a file name to open_dataset(), that works fine. But if you pass in a file-like object (as is done here with fsspec) the file-like object needs to be serializable.

 
{
    "total_count": 2,
    "+1": 2,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Serialization issue with distributed, h5netcdf, and fsspec (ImplicitToExplicitIndexingAdapter) 745801652
729863863 https://github.com/pydata/xarray/issues/4591#issuecomment-729863863 https://api.github.com/repos/pydata/xarray/issues/4591 MDEyOklzc3VlQ29tbWVudDcyOTg2Mzg2Mw== rabernat 1197350 2020-11-18T18:15:16Z 2020-11-18T18:15:16Z MEMBER

Thanks for your quick response to this Martin!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Serialization issue with distributed, h5netcdf, and fsspec (ImplicitToExplicitIndexingAdapter) 745801652
729837649 https://github.com/pydata/xarray/issues/4591#issuecomment-729837649 https://api.github.com/repos/pydata/xarray/issues/4591 MDEyOklzc3VlQ29tbWVudDcyOTgzNzY0OQ== shoyer 1217238 2020-11-18T17:37:58Z 2020-11-18T17:37:58Z MEMBER

H5NetCDFArrayWrapper is definitely supposed to be serializable with dask -- that's one of main reasons why these array wrapper classes exist in the first place.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Serialization issue with distributed, h5netcdf, and fsspec (ImplicitToExplicitIndexingAdapter) 745801652
729796223 https://github.com/pydata/xarray/issues/4591#issuecomment-729796223 https://api.github.com/repos/pydata/xarray/issues/4591 MDEyOklzc3VlQ29tbWVudDcyOTc5NjIyMw== rabernat 1197350 2020-11-18T16:31:14Z 2020-11-18T16:31:14Z MEMBER

Can you figure out how the http version differs from the gcs version? That might hold a clue.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Serialization issue with distributed, h5netcdf, and fsspec (ImplicitToExplicitIndexingAdapter) 745801652
729793908 https://github.com/pydata/xarray/issues/4591#issuecomment-729793908 https://api.github.com/repos/pydata/xarray/issues/4591 MDEyOklzc3VlQ29tbWVudDcyOTc5MzkwOA== rabernat 1197350 2020-11-18T16:27:30Z 2020-11-18T16:27:30Z MEMBER

I finally found a permutation that works, which makes me think this is an fsspec error.

```python import gcsfs

gcs = gcsfs.GCSFileSystem() url = 'gs://ldeo-glaciology/bedmachine/BedMachineAntarctica_2019-11-05_v01.nc'
openfile = gcs.open(url, mode='rb') dsgcs = xr.open_dataset(openfile, chunks=3000) dsgcs.surface.mean().compute() ```

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Serialization issue with distributed, h5netcdf, and fsspec (ImplicitToExplicitIndexingAdapter) 745801652

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);