home / github

Menu
  • Search all tables
  • GraphQL API

issue_comments

Table actions
  • GraphQL API for issue_comments

10 rows where issue = 787947436 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 3

  • kmuehlbauer 6
  • yt87 3
  • dcherian 1

author_association 2

  • MEMBER 7
  • NONE 3

issue 1

  • h5netcdf fails to decode attribute coordinates. · 10 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1081886061 https://github.com/pydata/xarray/issues/4822#issuecomment-1081886061 https://api.github.com/repos/pydata/xarray/issues/4822 IC_kwDOAMm_X85AfEVt dcherian 2448579 2022-03-29T13:39:45Z 2022-03-29T13:39:45Z MEMBER

Thanks @kmuehlbauer .

@yt87 please reopen if you still have trouble after upgrading h5netcdf

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  h5netcdf fails to decode attribute coordinates. 787947436
1081745934 https://github.com/pydata/xarray/issues/4822#issuecomment-1081745934 https://api.github.com/repos/pydata/xarray/issues/4822 IC_kwDOAMm_X85AeiIO kmuehlbauer 5821660 2022-03-29T11:20:44Z 2022-03-29T11:20:44Z MEMBER

@yt87 This works for me with latest h5netcdf version. There have been several fixes regarding attribute reading. The issue might be closed after confirming.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  h5netcdf fails to decode attribute coordinates. 787947436
762725640 https://github.com/pydata/xarray/issues/4822#issuecomment-762725640 https://api.github.com/repos/pydata/xarray/issues/4822 MDEyOklzc3VlQ29tbWVudDc2MjcyNTY0MA== kmuehlbauer 5821660 2021-01-19T09:43:53Z 2021-01-19T09:43:53Z MEMBER

Possibly related issue: https://github.com/h5netcdf/h5netcdf/issues/27

Anyway, I think I've found the root-cause of the issue. So with netcdf4-python it is possible to specify writing of attributes as NC_STRING string attributes.

The following will create a netcdf file which is readable with current xarray and h5netcdf backend.

```python import netCDF4 as nc import xarray as xr rootgrp = nc.Dataset("test_nc4_attrs.nc", mode="w", format="NETCDF4")

rootgrp.set_ncstring_attrs(True)

x = rootgrp.createDimension("x", 1) y = rootgrp.createDimension("y", 1) foo = rootgrp.createVariable("foo", "i4", ("y", "x")) foo.coordinates = "y x" foo[0, 0] = 0 rootgrp.close() ds = xr.open_dataset("test_nc4_attrs.nc", engine="h5netcdf") print(ds) print(ds.foo) ds.close() ```

If you uncomment the one line, this creates netcdf file which breaks with the error @yt87 showed above. If you try to open the file with decode_cf=False and write it back using to_netcdf the same error occurs.

h5dump before uncomment ATTRIBUTE "coordinates" { DATATYPE H5T_STRING { STRSIZE 3; STRPAD H5T_STR_NULLTERM; CSET H5T_CSET_ASCII; CTYPE H5T_C_S1; } DATASPACE SCALAR DATA { (0): "y x" } }

h5dump after uncomment ATTRIBUTE "coordinates" { DATATYPE H5T_STRING { STRSIZE H5T_VARIABLE; STRPAD H5T_STR_NULLTERM; CSET H5T_CSET_UTF8; CTYPE H5T_C_S1; } DATASPACE SIMPLE { ( 1 ) / ( 1 ) } DATA { (0): "y x" } }

Another thing I found, that this only accounts for coordinate attributes. All other attributes are read correctly AFAIKT. So with this in mind I would consider that a bug. The assumption that attributes are always scalar doesn't hold true.

And one more thing, also roundtripping in netcdf4 does not keep the NC_STRING type:

```python import netCDF4 as nc import xarray as xr rootgrp = nc.Dataset("test_nc4_attrs.nc", mode="w", format="NETCDF4") rootgrp.set_ncstring_attrs(True) x = rootgrp.createDimension("x", 1) y = rootgrp.createDimension("y", 1) foo = rootgrp.createVariable("foo", "i4", ("y", "x")) rootgrp.coordinates = "z y x" foo.coordinates = "y x" foo[0, 0] = 0 rootgrp.close()

ds = xr.open_dataset("test_nc4_attrs.nc", engine="netcdf4") ds.to_netcdf("test_nc4_attrs_out.nc") ds.close() ```

``` ncdump test_nc4_attrs.nc

netcdf test_nc4_attrs { dimensions: x = 1 ; y = 1 ; variables: int foo(y, x) ; string foo:coordinates = "y x" ;

// global attributes: string :coordinates = "z y x" ; data:

foo = 0 ; } ```

``` ncdump test_nc4_attrs_out.nc

netcdf test_nc4_attrs_out { dimensions: y = 1 ; x = 1 ; variables: int foo(y, x) ; foo:coordinates = "y x" ; data:

foo = 0 ; } ```

So changes are needed in several places to check if the attribute is scalar or array, not only for coordinates attributes. Thoughts?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  h5netcdf fails to decode attribute coordinates. 787947436
762438483 https://github.com/pydata/xarray/issues/4822#issuecomment-762438483 https://api.github.com/repos/pydata/xarray/issues/4822 MDEyOklzc3VlQ29tbWVudDc2MjQzODQ4Mw== yt87 40218891 2021-01-18T19:39:34Z 2021-01-18T19:39:34Z NONE

You might be right. Adding -k nc4 works when string is removed from attribute specification. If it is present, the error is as before: AttributeError: 'numpy.ndarray' object has no attribute 'split'.

However, after changing my AWS script to ``` import s3fs import xarray as xr

s3 = s3fs.S3FileSystem(anon=True) s3path = 's3://wrf-se-ak-ar5/gfdl/hist/daily/1988/WRFDS_1988-04-23.nc'

ds = xr.open_dataset(s3.open(s3path), engine='scipy') print(ds) ` the error isTypeError: Error: None is not a valid NetCDF 3 file``.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  h5netcdf fails to decode attribute coordinates. 787947436
762429899 https://github.com/pydata/xarray/issues/4822#issuecomment-762429899 https://api.github.com/repos/pydata/xarray/issues/4822 MDEyOklzc3VlQ29tbWVudDc2MjQyOTg5OQ== kmuehlbauer 5821660 2021-01-18T19:18:22Z 2021-01-18T19:18:22Z MEMBER

@yt87 Thanks for providing all this.

Could you please give -k 'nc4' as option to the call to ncgen and report back? I think this is a problem with data model.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  h5netcdf fails to decode attribute coordinates. 787947436
762423707 https://github.com/pydata/xarray/issues/4822#issuecomment-762423707 https://api.github.com/repos/pydata/xarray/issues/4822 MDEyOklzc3VlQ29tbWVudDc2MjQyMzcwNw== yt87 40218891 2021-01-18T19:03:19Z 2021-01-18T19:03:19Z NONE

This is how I did it:

``` $ ncdump /tmp/x.nc netcdf x { dimensions: x = 1 ; y = 1 ; variables: int foo(y, x) ; foo:coordinates = "x y" ; data:

foo = 0 ; } $ rm x.nc $ ncgen -o x.nc < x.cdl $ python -c "import xarray as xr; ds = xr.open_dataset('/tmp/x.nc', engine='h5netcdf'); print(ds)" ``` Engine netcdf4 works fine, with string or without.

My original code retrieving data from AWS: ``` import s3fs import xarray as xr

s3 = s3fs.S3FileSystem(anon=True) s3path = 's3://wrf-se-ak-ar5/gfdl/hist/daily/1988/WRFDS_1988-04-23.nc'

ds = xr.open_dataset(s3.open(s3path)) print(ds) ` Addingdecode_cf=False`` is a workaround. All attributes are arrays:

Attributes: contact: ['rtladerjr@alaska.edu'] info: ['Alaska CASC'] data: ['Downscaled GFDL-CM3'] format: ['version 2'] date: ['Mon Jul 1 15:17:16 AKDT 2019']

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  h5netcdf fails to decode attribute coordinates. 787947436
762400927 https://github.com/pydata/xarray/issues/4822#issuecomment-762400927 https://api.github.com/repos/pydata/xarray/issues/4822 MDEyOklzc3VlQ29tbWVudDc2MjQwMDkyNw== kmuehlbauer 5821660 2021-01-18T18:07:27Z 2021-01-18T18:07:27Z MEMBER

If you want a quick fix to read the file, add kwarg decode_cf=False in open_dataset. Then fix the coordinate attribute and call xr.decode_cf(ds).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  h5netcdf fails to decode attribute coordinates. 787947436
762396972 https://github.com/pydata/xarray/issues/4822#issuecomment-762396972 https://api.github.com/repos/pydata/xarray/issues/4822 MDEyOklzc3VlQ29tbWVudDc2MjM5Njk3Mg== kmuehlbauer 5821660 2021-01-18T17:58:07Z 2021-01-18T17:58:07Z MEMBER

Hmm, this worked in my case. How exactly did you create the netcdf file?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  h5netcdf fails to decode attribute coordinates. 787947436
762376418 https://github.com/pydata/xarray/issues/4822#issuecomment-762376418 https://api.github.com/repos/pydata/xarray/issues/4822 MDEyOklzc3VlQ29tbWVudDc2MjM3NjQxOA== yt87 40218891 2021-01-18T17:12:53Z 2021-01-18T17:19:53Z NONE

Dropping string changes error to Unable to open file (file signature not found). This issue popped up while reading data from https://registry.opendata.aws/wrf-se-alaska-snap/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  h5netcdf fails to decode attribute coordinates. 787947436
762258597 https://github.com/pydata/xarray/issues/4822#issuecomment-762258597 https://api.github.com/repos/pydata/xarray/issues/4822 MDEyOklzc3VlQ29tbWVudDc2MjI1ODU5Nw== kmuehlbauer 5821660 2021-01-18T13:42:20Z 2021-01-18T13:42:20Z MEMBER

@yt87 AFAIK the CDL doesn't declare attribute types specifically. This should work:

``` netcdf x { dimensions: x = 1 ; y = 1 ; variables: int foo(y, x) ; foo:coordinates = "x y" ; data:

foo = 0 ; } ```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  h5netcdf fails to decode attribute coordinates. 787947436

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);
Powered by Datasette · Queries took 641.307ms · About: xarray-datasette