issue_comments

Hi, second test case indicated by Apatlpo on on 12 Jul 2018, brakes ```python nx, ny, nt = 32, 32, 64 ds = xr.Dataset({}, coords={'x':np.arange(nx),'y':np.arange(ny), 't': np.arange(nt)}) ds = ds.assign(v=ds.tnp.cos(np.pi/180./100ds.x)np.cos(np.pi/180./50ds.y)) ds = ds.chunk({'t': 1, 'x': nx/2, 'y': ny/2}) ds.to_zarr('data.zarr', mode='w')

```

python ds = xr.open_zarr('data.zarr') ds = ds.chunk({'t': nt, 'x': nx/4, 'y': ny/4}) ds.to_zarr('data_rechunked.zarr', mode='w')

Err message is following . ValueError: Final chunk of Zarr array must be the same size or smaller than the first. The specified Zarr chunk encoding is (1, 16, 16), but (64,) in variable Dask chunks ((64,), (8, 8, 8, 8), (8, 8, 8, 8)) is incompatible. Consider rechunking using `chunk()

(if I add del ds.v.encoding['chunks'] as follows, it does not break) python nx, ny, nt = 32, 32, 64 ds = xr.Dataset({}, coords={'x':np.arange(nx),'y':np.arange(ny), 't': np.arange(nt)}) ds = ds.assign(v=ds.t*np.cos(np.pi/180./100*ds.x)*np.cos(np.pi/180./50*ds.y)) ds = ds.chunk({'t': 1, 'x': nx/2, 'y': ny/2}) ds.to_zarr('data.zarr', mode='w') ds = xr.open_zarr('data.zarr') del ds.v.encoding['chunks'] ds = ds.chunk({'t': nt, 'x': nx/4, 'y': ny/4}) ds.to_zarr('data_rechunked.zarr', mode='w')

https://github.com/pydata/xarray/blob/64a7d1144c78eacbcd2401d0aa06e86f4047b0a7/xarray/backends/netCDF4_.py#L208-L209

Could you please be more specific about where this is done for netCDF?

We do Ryan's option 2 for netCDF files and it works pretty well. On Thu, Jul 12, 2018 at 8:24 AM Ryan Abernathey notifications@github.com wrote:

Yes, this is the same underlying issue.

On Thu, Jul 12, 2018 at 2:59 PM Aurélien Ponte notifications@github.com wrote:

Note that there is also a fix here that is simply del ds['v'].encoding['chunks'] prior to data storage.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/2278#issuecomment-404503718, or mute the thread < https://github.com/notifications/unsubscribe-auth/ABJFJp-x0xW1Pe_zzEmnO41Ae3tYE541ks5uF0hAgaJpZM4VK7Q0

.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/2278#issuecomment-404510872, or mute the thread https://github.com/notifications/unsubscribe-auth/ABKS1tIqTavCwrLu9qcwDQtb98SG6G_Tks5uF04kgaJpZM4VK7Q0 .

Yes, this is the same underlying issue.

On Thu, Jul 12, 2018 at 2:59 PM Aurélien Ponte notifications@github.com wrote:

Note that there is also a fix here that is simply del ds['v'].encoding['chunks'] prior to data storage.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/2278#issuecomment-404503718, or mute the thread https://github.com/notifications/unsubscribe-auth/ABJFJp-x0xW1Pe_zzEmnO41Ae3tYE541ks5uF0hAgaJpZM4VK7Q0 .

Note that there is also a fix for case 2 that is simply del ds['v'].encoding['chunks'] prior to data storage.

With the same case, I have another error message which may reflect the same issue (or not), maybe you can tell me. The error message is different which is the reason I am posting this.

Starting from the same dataset: nx, ny, nt = 32, 32, 64 ds = xr.Dataset({}, coords={'x':np.arange(nx),'y':np.arange(ny), 't': np.arange(nt)}) ds = ds.assign(v=ds.t*np.cos(np.pi/180./100*ds.x)*np.cos(np.pi/180./50*ds.y)) ds = ds.chunk({'t': 1, 'x': nx/2, 'y': ny/2}) ds.to_zarr('data.zarr', mode='w')

Case 1 works fine: ds = ds.chunk({'t': nt, 'x': nx/4, 'y': ny/4}) ds.to_zarr('data_rechunked.zarr', mode='w')

Case 2 breaks: ds = xr.open_zarr('data.zarr') ds = ds.chunk({'t': nt, 'x': nx/4, 'y': ny/4}) ds.to_zarr('data_rechunked.zarr', mode='w') with the following error message: .... NotImplementedError: Specified zarr chunks (1, 16, 16) would overlap multiple dask chunks ((64,), (8, 8, 8, 8), (8, 8, 8, 8)). This is not implemented in xarray yet. Consider rechunking the data using `chunk()` or specifying different chunks in encoding.

Any idea about how serious this is and/or where it's coming from?

The source of the bug is that encoding metadata chunks (which describes the chunk size of the underlying zarr store) is automatically getting populated when you load the zarr store (ds = xr.open_zarr('data.zarr')), and this encoding metadata is being preserved as you transform (sub-select) the dataset. Some possible solutions would be to

1. Not put chunks into encoding at all.
2. Figure out a way to strip chunks when performing selection operations or other operations that change shape.

Idea 1 is easier but would mean discarding some relevant metadata about encoding. This would break round-tripping of the un-modified zarr dataset.

thanks for the workaround suggestion. Apparently you also need to delete chunks for the t singleton coordinate though. The workaround looks at the end like: ds = xr.open_zarr('data.zarr') del ds['v'].encoding['chunks'] del ds['t'].encoding['chunks'] ds.isel(t=0).to_zarr('data_t0.zarr', mode='w') Any idea about how serious this is and/or where it's coming from?

Yes, this is definitely a bug.

One workaround is to explicitly remove the broken chunks encoding from the loaded dataset, e.g., del ds['v'].encoding['chunks']