issue_comments

Table actions

3 rows where issue = 1605108888 and user = 5821660 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

  • kmuehlbauer · 3 ✖

issue 1

  • xr.open_mfdataset doesn't work with fsspec and dask · 3 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1466263309 https://github.com/pydata/xarray/issues/7574#issuecomment-1466263309 https://api.github.com/repos/pydata/xarray/issues/7574 IC_kwDOAMm_X85XZWcN kmuehlbauer 5821660 2023-03-13T14:37:40Z 2023-03-13T14:37:40Z MEMBER

Thanks @martindurant for looking into this. I'll try to go to the bottom of this, if I find the time.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   xr.open_mfdataset doesn't work with fsspec and dask 1605108888
1463576675 https://github.com/pydata/xarray/issues/7574#issuecomment-1463576675 https://api.github.com/repos/pydata/xarray/issues/7574 IC_kwDOAMm_X85XPGhj kmuehlbauer 5821660 2023-03-10T10:12:27Z 2023-03-10T10:12:27Z MEMBER

The difference between parallel=False vs parallel=True is how the cache is setup:

  • parallel=False, the cache-key looks like:

[<class 'h5netcdf.core.File'>, (<File-like object S3FileSystem, noaa-goes16/ABI-L2-LSTC/2022/185/03/OR_ABI-L2-LSTC-M6_G16_s20221850301180_e20221850303553_c20221850305091.nc>,), 'r', (('decode_vlen_strings', True), ('invalid_netcdf', None)), '50fa07a0-7db1-420d-b2e1-5ad4c1328be4']

  • parallel=True, the cache-key looks like:

[<class 'h5netcdf.core.File'>, ((b'FHIB\x00\x1dn"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00u[...cutted...],), 'r', (('decode_vlen_strings', True), ('invalid_netcdf', None)), 'be975f9d-710d-4c2e-9e03-606b6f8f0810']

So this has something to do, when and how the cache is initialized. In the first case we've got the file-like object, but in the second case we've got the binary string. @martindurant do you have some idea what's going on here and where to look for fixing this?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   xr.open_mfdataset doesn't work with fsspec and dask 1605108888
1463528885 https://github.com/pydata/xarray/issues/7574#issuecomment-1463528885 https://api.github.com/repos/pydata/xarray/issues/7574 IC_kwDOAMm_X85XO621 kmuehlbauer 5821660 2023-03-10T09:36:38Z 2023-03-10T09:36:38Z MEMBER

@Berhinj The issue is with parallel=True. The following works as expected:

```python import fsspec import xarray as xr paths = [ 's3://noaa-goes16/ABI-L2-LSTC/2022/185/03/OR_ABI-L2-LSTC-M6_G16_s20221850301180_e20221850303553_c20221850305091.nc', 's3://noaa-goes16/ABI-L2-LSTC/2022/185/02/OR_ABI-L2-LSTC-M6_G16_s20221850201180_e20221850203553_c20221850205142.nc' ]

fs = fsspec.filesystem('s3', anon=True) flist = [fs.open(path, mode="rb") for path in paths]

single dataset

ds = xr.open_dataset(flist[0], engine="h5netcdf") print(ds)

multiple datasets, ATT: parallel=False works

ts = xr.open_mfdataset( flist, engine="h5netcdf", combine="nested", concat_dim="t", parallel=False ) print(ts) ```

I've not digged further, but it looks like it has some issues with the file-object...

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   xr.open_mfdataset doesn't work with fsspec and dask 1605108888

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);