html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue https://github.com/pydata/xarray/issues/2278#issuecomment-493431087,https://api.github.com/repos/pydata/xarray/issues/2278,493431087,MDEyOklzc3VlQ29tbWVudDQ5MzQzMTA4Nw==,46813815,2019-05-17T12:11:21Z,2019-05-17T14:03:38Z,NONE,"Hi, second test case indicated by Apatlpo on on 12 Jul 2018, brakes ```python nx, ny, nt = 32, 32, 64 ds = xr.Dataset({}, coords={'x':np.arange(nx),'y':np.arange(ny), 't': np.arange(nt)}) ds = ds.assign(v=ds.t*np.cos(np.pi/180./100*ds.x)*np.cos(np.pi/180./50*ds.y)) ds = ds.chunk({'t': 1, 'x': nx/2, 'y': ny/2}) ds.to_zarr('data.zarr', mode='w') ``` ``` python ds = xr.open_zarr('data.zarr') ds = ds.chunk({'t': nt, 'x': nx/4, 'y': ny/4}) ds.to_zarr('data_rechunked.zarr', mode='w') ``` Err message is following . ``` ValueError: Final chunk of Zarr array must be the same size or smaller than the first. The specified Zarr chunk encoding is (1, 16, 16), but (64,) in variable Dask chunks ((64,), (8, 8, 8, 8), (8, 8, 8, 8)) is incompatible. Consider rechunking using `chunk() ``` (if I add del ds.v.encoding['chunks'] as follows, it does not break) ```python nx, ny, nt = 32, 32, 64 ds = xr.Dataset({}, coords={'x':np.arange(nx),'y':np.arange(ny), 't': np.arange(nt)}) ds = ds.assign(v=ds.t*np.cos(np.pi/180./100*ds.x)*np.cos(np.pi/180./50*ds.y)) ds = ds.chunk({'t': 1, 'x': nx/2, 'y': ny/2}) ds.to_zarr('data.zarr', mode='w') ds = xr.open_zarr('data.zarr') del ds.v.encoding['chunks'] ds = ds.chunk({'t': nt, 'x': nx/4, 'y': ny/4}) ds.to_zarr('data_rechunked.zarr', mode='w') ```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,340192831 https://github.com/pydata/xarray/issues/2278#issuecomment-404970837,https://api.github.com/repos/pydata/xarray/issues/2278,404970837,MDEyOklzc3VlQ29tbWVudDQwNDk3MDgzNw==,1217238,2018-07-13T22:37:23Z,2018-07-13T22:37:23Z,MEMBER,https://github.com/pydata/xarray/blob/64a7d1144c78eacbcd2401d0aa06e86f4047b0a7/xarray/backends/netCDF4_.py#L208-L209,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,340192831 https://github.com/pydata/xarray/issues/2278#issuecomment-404873326,https://api.github.com/repos/pydata/xarray/issues/2278,404873326,MDEyOklzc3VlQ29tbWVudDQwNDg3MzMyNg==,11750960,2018-07-13T15:48:46Z,2018-07-13T15:48:46Z,CONTRIBUTOR,Could you please be more specific about where this is done for netCDF?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,340192831 https://github.com/pydata/xarray/issues/2278#issuecomment-404530618,https://api.github.com/repos/pydata/xarray/issues/2278,404530618,MDEyOklzc3VlQ29tbWVudDQwNDUzMDYxOA==,1217238,2018-07-12T14:25:02Z,2018-07-12T14:25:02Z,MEMBER,"We do Ryan's option 2 for netCDF files and it works pretty well. On Thu, Jul 12, 2018 at 8:24 AM Ryan Abernathey wrote: > Yes, this is the same underlying issue. > > On Thu, Jul 12, 2018 at 2:59 PM Aurélien Ponte > wrote: > > > Note that there is also a fix here that is simply del > > ds['v'].encoding['chunks'] prior to data storage. > > > > — > > You are receiving this because you commented. > > Reply to this email directly, view it on GitHub > > , > or mute > > the thread > > < > https://github.com/notifications/unsubscribe-auth/ABJFJp-x0xW1Pe_zzEmnO41Ae3tYE541ks5uF0hAgaJpZM4VK7Q0 > > > > . > > > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > , or mute > the thread > > . > ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,340192831 https://github.com/pydata/xarray/issues/2278#issuecomment-404510872,https://api.github.com/repos/pydata/xarray/issues/2278,404510872,MDEyOklzc3VlQ29tbWVudDQwNDUxMDg3Mg==,1197350,2018-07-12T13:24:51Z,2018-07-12T13:24:51Z,MEMBER,"Yes, this is the same underlying issue. On Thu, Jul 12, 2018 at 2:59 PM Aurélien Ponte wrote: > Note that there is also a fix here that is simply del > ds['v'].encoding['chunks'] prior to data storage. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > , or mute > the thread > > . > ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,340192831 https://github.com/pydata/xarray/issues/2278#issuecomment-404503718,https://api.github.com/repos/pydata/xarray/issues/2278,404503718,MDEyOklzc3VlQ29tbWVudDQwNDUwMzcxOA==,11750960,2018-07-12T12:59:44Z,2018-07-12T13:00:01Z,CONTRIBUTOR,Note that there is also a fix for case 2 that is simply `del ds['v'].encoding['chunks']` prior to data storage.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,340192831 https://github.com/pydata/xarray/issues/2278#issuecomment-404503025,https://api.github.com/repos/pydata/xarray/issues/2278,404503025,MDEyOklzc3VlQ29tbWVudDQwNDUwMzAyNQ==,11750960,2018-07-12T12:57:17Z,2018-07-12T12:57:35Z,CONTRIBUTOR,"With the same case, I have another error message which may reflect the same issue (or not), maybe you can tell me. The error message is different which is the reason I am posting this. Starting from the same dataset: ``` nx, ny, nt = 32, 32, 64 ds = xr.Dataset({}, coords={'x':np.arange(nx),'y':np.arange(ny), 't': np.arange(nt)}) ds = ds.assign(v=ds.t*np.cos(np.pi/180./100*ds.x)*np.cos(np.pi/180./50*ds.y)) ds = ds.chunk({'t': 1, 'x': nx/2, 'y': ny/2}) ds.to_zarr('data.zarr', mode='w') ``` Case 1 works fine: ``` ds = ds.chunk({'t': nt, 'x': nx/4, 'y': ny/4}) ds.to_zarr('data_rechunked.zarr', mode='w') ``` Case 2 breaks: ``` ds = xr.open_zarr('data.zarr') ds = ds.chunk({'t': nt, 'x': nx/4, 'y': ny/4}) ds.to_zarr('data_rechunked.zarr', mode='w') ``` with the following error message: ``` .... NotImplementedError: Specified zarr chunks (1, 16, 16) would overlap multiple dask chunks ((64,), (8, 8, 8, 8), (8, 8, 8, 8)). This is not implemented in xarray yet. Consider rechunking the data using `chunk()` or specifying different chunks in encoding. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,340192831 https://github.com/pydata/xarray/issues/2278#issuecomment-404429223,https://api.github.com/repos/pydata/xarray/issues/2278,404429223,MDEyOklzc3VlQ29tbWVudDQwNDQyOTIyMw==,1197350,2018-07-12T08:15:43Z,2018-07-12T08:16:02Z,MEMBER,"> Any idea about how serious this is and/or where it's coming from? The source of the bug is that encoding metadata `chunks` (which describes the chunk size of the underlying zarr store) is automatically getting populated when you load the zarr store (`ds = xr.open_zarr('data.zarr')`), and this encoding metadata is being preserved as you transform (sub-select) the dataset. Some possible solutions would be to 1. Not put `chunks` into encoding at all. 2. Figure out a way to strip `chunks` when performing selection operations or other operations that change shape. Idea 1 is easier but would mean discarding some relevant metadata about encoding. This would break round-tripping of the un-modified zarr dataset.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,340192831 https://github.com/pydata/xarray/issues/2278#issuecomment-404415760,https://api.github.com/repos/pydata/xarray/issues/2278,404415760,MDEyOklzc3VlQ29tbWVudDQwNDQxNTc2MA==,11750960,2018-07-12T07:25:36Z,2018-07-12T07:25:36Z,CONTRIBUTOR,"thanks for the workaround suggestion. Apparently you also need to delete chunks for the `t` singleton coordinate though. The workaround looks at the end like: ``` ds = xr.open_zarr('data.zarr') del ds['v'].encoding['chunks'] del ds['t'].encoding['chunks'] ds.isel(t=0).to_zarr('data_t0.zarr', mode='w') ``` Any idea about how serious this is and/or where it's coming from?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,340192831 https://github.com/pydata/xarray/issues/2278#issuecomment-404277786,https://api.github.com/repos/pydata/xarray/issues/2278,404277786,MDEyOklzc3VlQ29tbWVudDQwNDI3Nzc4Ng==,1217238,2018-07-11T19:06:20Z,2018-07-11T19:06:20Z,MEMBER,"Yes, this is definitely a bug. One workaround is to explicitly remove the broken `chunks` encoding from the loaded dataset, e.g., `del ds['v'].encoding['chunks']`","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,340192831