issue_comments

Related issue in dask: https://github.com/dask/dask/issues/6363

This isn't a fix for the overhead required to manage an arbitrarily large graph, but note that creating chunks this small (size 1 in this case) is explicitly not recommended. See the dask docs on Array Best Practices: Select a good chunk size - they recommend chunks no smaller than 100 MB. Your chunks are 8 bytes. This creates 1 billion tasks, which does result in an enormous overhead - there's no way around this. Note that storing this on disk would not help - the problem results from the fact that 1 billion tasks will almost certainly overwhelm any dask scheduler. The general dask best practices guide recommends keeping the number of tasks below 1 million if possible.

Also, I don't think that the issue here is in specifying the universe of the tasks that need to be created, but rather in creating and managing the python task objects themselves. So pre-computing or storing them wouldn't help.

For me, changing to (1000, 1000, 100) chunks (~750MB for a float64 array) reduces the time to a couple ms: python In [16]: %%timeit ...: ...: chunks = (1000, 1000, 100) ...: ds = xr.Dataset(data_vars={ ...: "foo": (('x', 'y', 'z'), dask.array.empty((1000, 1000, 1000), chunks=(1000, 1000, 1000)))}) ...: ds.to_zarr(store='data', group='ds.zarr', compute=False, encoding={'foo': {'chunks': chunks}}, mode='w') ...: ds_loaded = xr.open_zarr(group='ds.zarr', store='data') ...: ...: 6.36 ms ± 111 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

With this chunking scheme, you could store and work with much, much more data. In fact, scaling the size of your example by 3 orders of magnitude only increases the runtime by ~5x: python In [18]: %%timeit ...: ...: chunks = (1000, 1000, 100, 1) ...: ds = xr.Dataset(data_vars={ ...: "foo": (('w', 'x', 'y', 'z'), dask.array.empty((1000, 1000, 1000, 1000), chunks=(1000, 1000, 1000, 1)))}) ...: ds.to_zarr(store='data', group='ds.zarr', compute=False, encoding={'foo': {'chunks': chunks}}, mode='w') ...: ds_loaded = xr.open_zarr(group='ds.zarr', store='data') ...: ...: 36.9 ms ± 2.23 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) So if re-writing your arrays with larger chunks is an option I think this could get around the problem you're seeing?

@dcherian thanks for your reply. I know Xarray can't do anything about the Dask computations of the chunks. My question was if it was possible to save the Dask chunk informations on the Zarr metadata such that it is not neccessary to calculate them ie. run the the getem() function from Dask that takes too long to run and increases memory.

Following example runs out of memory on my computer. I have 16 GB RAM.

```python

import dask import xarray as xr

chunks = (1, 1, 1) ds = xr.Dataset(data_vars={ "foo": (('x', 'y', 'z'), dask.array.empty((1000, 1000, 1000), chunks=(1000, 1000, 1000)))}) ds.to_zarr(store='data', group='ds.zarr', compute=False, encoding={'foo': {'chunks': chunks}}) ds_loaded = xr.open_zarr(group='ds.zarr', store='data')

```

@DonjetaR Thanks for the very well written issue! and for confirming #6013.

Could you please add a minimum reproducible example to #6013? I think that would help greatly

The following runs in 1s for me which seems OK. Can you open an issue over at dask about this. Xarray can't do anything about it. ``` python import dask.array

dim_size = (10, 15_000, 15_000) chunks = dask.array.empty(dim_size, chunks=(10, 10, 10)).chunks

%timeit dask.array.core.slices_from_chunks(chunks) ``` On repeated runs it drops down to 200ms (because of caching I guess), so it was important to restart the kernel to test it out.