issues: 662982199

I have a large xarray.Dataset stored as a zarr. I want to perform some custom operations on it that cannot be done by just using numpy-like functions that a Dask cluster will automatically deal with. Therefore, I partition the dataset into small subsets and for each subset submit to my Dask cluster a task of the form def my_task(zarr_path, subset_index): ds = xarray.open_zarr(zarr_path) # this returns an xarray.Dataset containing a dask.array sel = ds.sel(partition_index) sel = sel.load() # I want to get the data into memory # then do my custom operations ... However, I have noticed this creates a "task within a task": when a worker receives "my_task", it in turn submits tasks to the cluster to load the relevant part of the dataset. To avoid this and ensure that the full task is executed within the worker, I am submitting instead the task: def my_task_2(zarr_path, subset_index): with dask.config.set(scheduler="threading"): my_task(zarr_path, subset_index) Is this the best way to do this? What's the best practice for this kind of situation?

I have already posted this on stackoverflow but did not get any answer, so I am adding this here hoping it increases visibility. Apologies if this is considered "pollution". https://stackoverflow.com/questions/62874267/parallel-tasks-on-subsets-of-a-dask-array-wrapped-in-an-xarray-dataset