issues: 884209406
This data as json
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | active_lock_reason | draft | pull_request | body | reactions | performed_via_github_app | state_reason | repo | type |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
884209406 | MDU6SXNzdWU4ODQyMDk0MDY= | 5286 | Zarr chunks would overlap multiple dask chunks error | 6130352 | closed | 0 | 3 | 2021-05-10T13:20:46Z | 2021-05-12T16:16:05Z | 2021-05-12T16:16:05Z | NONE | Would it be possible to get an explanation on how this situation results in a zarr chunk overlapping multiple dask chunks? This code below is generating an array with 2 chunks, selecting one row from each chunk, and then writing that resulting two row array back to zarr. I don't see how it's possible in this case for one zarr chunk to correspond to different dask chunks. There are clearly two resulting dask chunks, two input zarr chunks, and a correspondence between them that should be 1 to 1 ... what does this error message really mean then? ```python import xarray as xr import dask.array as da ds = xr.Dataset(dict( x=(('a', 'b'), da.ones(shape=(10, 10), chunks=(5, 10))), )).assign(a=list(range(10))) ds <xarray.Dataset>Dimensions: (a: 10, b: 10)Coordinates:* a (a) int64 0 1 2 3 4 5 6 7 8 9Dimensions without coordinates: bData variables:x (a, b) float64 dask.array<chunksize=(5, 10), meta=np.ndarray>Write the dataset out!rm -rf /tmp/test.zarr ds.to_zarr('/tmp/test.zarr') Read it back in, subset to 1 record in two different chunks (two rows total), write back out!rm -rf /tmp/test2.zarr xr.open_zarr('/tmp/test.zarr').sel(a=[0, 11]).to_zarr('/tmp/test2.zarr') NotImplementedError: Specified zarr chunks encoding['chunks']=(5, 10) for variable named 'x' would overlap multiple dask chunks ((1, 1), (10,)). Writing this array in parallel with dask could lead to corrupted data. Consider either rechunking using
|
{ "url": "https://api.github.com/repos/pydata/xarray/issues/5286/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | 13221727 | issue |