home / github / issues

Menu
  • Search all tables
  • GraphQL API

issues: 446016536

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
446016536 MDU6SXNzdWU0NDYwMTY1MzY= 2974 Problems reading grouped netCDF file generated with h5netcdf engine 1991007 closed 0     3 2019-05-20T09:35:54Z 2019-05-21T07:13:01Z 2019-05-21T03:08:11Z NONE      

Code Sample, a copy-pastable example if possible

```python import xarray as xr

data1 = [[1, 2], [3, 4]] y1 = [1, 2] x1 = [1, 2]

data2 = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] y2 = [1, 2, 3] x2 = [1, 2, 3]

dataset1 = xr.Dataset({'data1': xr.DataArray(data1, dims=('y', 'x'), coords={'y': y1, 'x': x1})}) dataset2 = xr.Dataset({'data2': xr.DataArray(data2, dims=('y', 'x'), coords={'y': y2, 'x': x2})}) dataset1.to_netcdf('test.nc', mode='w', group='grp1', engine='h5netcdf') dataset2.to_netcdf('test.nc', mode='a', group='grp2', engine='h5netcdf')

xr.open_dataset('test.nc', group='grp1', engine='h5netcdf') # works xr.open_dataset('test.nc', group='grp1', engine='netcdf4') # fails ```

Traceback:

``` Traceback (most recent call last): File "test_xr.py", line 21, in <module> print(xr.open_dataset('test.nc', group='grp1', engine='netcdf4')) File "/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/h5netcdf/lib/python3.7/site-packages/xarray/backends/api.py", line 394, in open_dataset ds = maybe_decode_store(store) File "/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/h5netcdf/lib/python3.7/site-packages/xarray/backends/api.py", line 324, in maybe_decode_store drop_variables=drop_variables, use_cftime=use_cftime) File "/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/h5netcdf/lib/python3.7/site-packages/xarray/conventions.py", line 480, in decode_cf ds = Dataset(vars, attrs=attrs) File "/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/h5netcdf/lib/python3.7/site-packages/xarray/core/dataset.py", line 383, in __init__ self._set_init_vars_and_dims(data_vars, coords, compat) File "/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/h5netcdf/lib/python3.7/site-packages/xarray/core/dataset.py", line 405, in _set_init_vars_and_dims data_vars, coords, compat=compat) File "/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/h5netcdf/lib/python3.7/site-packages/xarray/core/merge.py", line 377, in merge_data_and_coords indexes=indexes) File "/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/h5netcdf/lib/python3.7/site-packages/xarray/core/merge.py", line 446, in merge_core expanded = expand_variable_dicts(aligned) File "/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/h5netcdf/lib/python3.7/site-packages/xarray/core/merge.py", line 222, in expand_variable_dicts var = as_variable(var, name=name) File "/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/h5netcdf/lib/python3.7/site-packages/xarray/core/variable.py", line 117, in as_variable obj = obj.to_index_variable() File "/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/h5netcdf/lib/python3.7/site-packages/xarray/core/variable.py", line 408, in to_index_variable encoding=self._encoding, fastpath=True) File "/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/h5netcdf/lib/python3.7/site-packages/xarray/core/variable.py", line 1825, in __init__ self._data = PandasIndexAdapter(self._data) File "/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/h5netcdf/lib/python3.7/site-packages/xarray/core/indexing.py", line 1219, in __init__ self.array = utils.safe_cast_to_index(array) File "/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/h5netcdf/lib/python3.7/site-packages/xarray/core/utils.py", line 78, in safe_cast_to_index index = pd.Index(np.asarray(array), **kwargs) File "/cmsaf/nfshome/routcm/Modules_CentOS/python/3.7.2/lib/python3.7/site-packages/numpy/core/numeric.py", line 501, in asarray return array(a, dtype, copy=False, order=order) File "/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/h5netcdf/lib/python3.7/site-packages/xarray/core/indexing.py", line 510, in __array__ return np.asarray(array[self.key], dtype=None) File "/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/h5netcdf/lib/python3.7/site-packages/xarray/backends/netCDF4_.py", line 64, in __getitem__ self._getitem) File "/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/h5netcdf/lib/python3.7/site-packages/xarray/core/indexing.py", line 778, in explicit_indexing_adapter result = raw_indexing_method(raw_key.tuple) File "/cmsaf/cmsaf-ops3/sfinkens/virtualenvs/h5netcdf/lib/python3.7/site-packages/xarray/backends/netCDF4_.py", line 75, in _getitem array = getitem(original_array, key) File "netCDF4/_netCDF4.pyx", line 4317, in netCDF4._netCDF4.Variable.__getitem__ File "netCDF4/_netCDF4.pyx", line 5249, in netCDF4._netCDF4.Variable._get File "netCDF4/_netCDF4.pyx", line 1842, in netCDF4._netCDF4._ensure_nc_success RuntimeError: NetCDF: Start+count exceeds dimension bound ```

Output of ncdump test.nc ``` ncdump test.nc netcdf test {

group: grp1 { dimensions: y = 3 ; x = 3 ; variables: int64 data1(y, x) ; int64 x(x) ; int64 y(y) ; data:

data1 = NetCDF: Start+count exceeds dimension bound Location: file vardata.c; line 478 ```

Problem description

If datasets with different coordinates have been written to different netCDF groups using the h5netcdf engine, the generated file cannot be read by ncdump or the netcdf4 engine. The file is readable by the h5netcdf engine, though.

If the file is generated using the netcdf4 engine, it is readable by both ncdump and the netcdf4/h5netcdf engines. Also, if the datasets have identical coordinates, there is no such problem.

Expected Output

Since the file is readable by the h5netcdf engine, I'm not sure whether this is actually a bug. But at least I would expect the output file to be readable by ncdump - independent of the engine.

Output of xr.show_versions()

``` commit: None python: 3.7.2 (default, Mar 15 2019, 15:45:45) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] python-bits: 64 OS: Linux OS-release: 3.10.0-957.12.1.el7.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: en_GB.UTF-8 libhdf5: 1.10.4 libnetcdf: 4.6.2 xarray: 0.12.1 pandas: 0.24.2 numpy: 1.15.4 scipy: 1.2.1 netCDF4: 1.4.3.2 pydap: None h5netcdf: 0.7.1 h5py: 2.9.0 Nio: None zarr: None cftime: 1.0.3.4 nc_time_axis: None PseudonetCDF: None rasterio: 1.0.22 cfgrib: None iris: None bottleneck: 1.2.1 dask: 1.1.4 distributed: 1.26.0 matplotlib: 3.0.3 cartopy: 0.17.0 seaborn: 0.9.0 setuptools: 40.6.2 pip: 18.1 conda: None pytest: 4.3.1 IPython: 7.3.0 sphinx: 1.8.5 ```
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/2974/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed 13221727 issue

Links from other tables

  • 0 rows from issues_id in issues_labels
  • 3 rows from issue in issue_comments
Powered by Datasette · Queries took 4.262ms · About: xarray-datasette