home / github / issues

Menu
  • Search all tables
  • GraphQL API

issues: 336273865

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
336273865 MDU6SXNzdWUzMzYyNzM4NjU= 2254 Writing Datasets to netCDF4 with "inconsistent" chunks 1554921 closed 0     3 2018-06-27T15:15:02Z 2018-06-29T05:07:27Z 2018-06-29T05:07:27Z CONTRIBUTOR      

Code Sample

```python import xarray as xr from dask.array import zeros, ones

Construct two variables with the same dimensions, but different chunking

x = zeros((100,100),dtype='f4',chunks=(50,100)) x = xr.DataArray(data=x, dims=('lat','lon'), name='x') y = ones((100,100),dtype='f4',chunks=(100,50)) y = xr.DataArray(data=y, dims=('lat','lon'), name='y')

Put them both into the same dataset

dset = xr.Dataset({'x':x,'y':y})

Save to a netCDF4 file.

dset.to_netcdf("test.nc") ```

The last line results in ValueError: inconsistent chunks

Problem description

This error is triggered by xarray.backends.api.to_netcdf's use of the dataset.chunks property in two places:

https://github.com/pydata/xarray/blob/bb581ca206c80eea80270ba508ec80ae0cd3941f/xarray/backends/api.py#L703

https://github.com/pydata/xarray/blob/bb581ca206c80eea80270ba508ec80ae0cd3941f/xarray/backends/api.py#L709

I'm assuming to_netcdf only needs to know if chunks are being used, not necessarily if they're consistent?

If I define a more general check python have_chunks = any(v.chunks for v in dataset.variables.values()) and replace the instances of dataset.chunks with have_chunks, then the netCDF4 file gets written without any problems (although the data seems to be stored contiguously instead of chunked).

Is this change as straight-forward as I think, or Is there something intrinsic about xarray.Dataset objects or writing to netCDF4 that require consistent chunks?

Output of xr.show_versions()

commit: bb581ca206c80eea80270ba508ec80ae0cd3941f python: 2.7.12.final.0 python-bits: 64 OS: Linux OS-release: 4.4.0-128-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: None.None xarray: 0.10.7 pandas: 0.23.1 numpy: 1.14.5 scipy: None netCDF4: 1.4.0 h5netcdf: None h5py: None Nio: None zarr: None bottleneck: None cyordereddict: None dask: 0.17.5 distributed: None matplotlib: None cartopy: None seaborn: None setuptools: 39.2.0 pip: 10.0.1 conda: None pytest: None IPython: None sphinx: None
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/2254/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed 13221727 issue

Links from other tables

  • 0 rows from issues_id in issues_labels
  • 3 rows from issue in issue_comments
Powered by Datasette · Queries took 0.83ms · About: xarray-datasette