• [x] Closes https://github.com/pydata/xarray/issues/5902
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

What happened:

Calling DataArray.unstack() spends time allocating an object-dtype NumPy array from values of the pandas MultiIndex.

What you expected to happen:

Faster unstack.

Minimal Complete Verifiable Example:

```python import pandas as pd import numpy as np import xarray as xr

t = pd.date_range("2000", periods=2) x = np.arange(1000) y = np.arange(1000) component = np.arange(4)

idx = pd.MultiIndex.from_product([t, y, x], names=["time", "y", "x"])

data = np.random.uniform(size=(len(idx), len(component))) arr = xr.DataArray( data, coords={"pixel": xr.DataArray(idx, name="pixel", dims="pixel"), "component": xr.DataArray(component, name="component", dims="component")}, dims=("pixel", "component") )

%time _ = arr.unstack() CPU times: user 6.33 s, sys: 295 ms, total: 6.62 s Wall time: 6.62 s ```

Anything else we need to know?:

For this example, >99% of the time is spent at on this line: https://github.com/pydata/xarray/blob/df7646182b17d829fe9b2199aebf649ddb2ed480/xarray/core/dataset.py#L4162, specifically on the call to v.data for the pixel array, which is a pandas MultiIndex.

Just going by the comments, it does seem like accessing v.data is necessary to perform the check. I'm wonder if we could make is_duck_dask_array a bit smarter, to avoid unnecessarily allocating data?

Alternatively, if that's too difficult, perhaps we could add a flag to unstack to disable those checks and just take the "slow" path. In my actual use-case, the slow _unstack_full_reindex is necessary since I have large Dask Arrays. But even then, the unstack completes in less than 3s, while I was getting OOM killed on the v.data checks.

Environment:

What happened:

When calling DataArray.mean on an all-NaN dataset, a warning is emitted if and only if a Dask array is used.

What you expected to happen:

Identical behavior between the two, probably no warning .

Minimal Complete Verifiable Example:

```python In [7]: import xarray as xr

In [8]: import numpy as np

In [9]: import dask.array as da

In [10]: import xarray as xr

In [11]: a = xr.DataArray(da.from_array(np.full((10, 10), np.nan)))

In [12]: a.mean(dim="dim_0").compute() /home/taugspurger/miniconda3/envs/tmp-adlfs/lib/python3.8/site-packages/dask/array/numpy_compat.py:39: RuntimeWarning: invalid value encountered in true_divide x = np.divide(x1, x2, out) Out[12]: <xarray.DataArray 'array-395d894c4e4d4ca165a189736da1f52d' (dim_1: 10)> array([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan]) Dimensions without coordinates: dim_1

In [13]: a.compute().mean(dim="dim_0") Out[13]: <xarray.DataArray 'array-395d894c4e4d4ca165a189736da1f52d' (dim_1: 10)> array([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan]) Dimensions without coordinates: dim_1 ```

Anything else we need to know?:

I haven't looked closely at why this is happening (I couldn't immediately find where .mean is reduced). I know that Dask has had some issues in the past where NumPy warnings filters are set during graph construction time, but aren't set when the graph is actually computed.

Environment:

What happened:

When reading an zarr dataset where the last chunk is smaller than the chunk size, users see a UserWarning that this may be inefficient, since the chunking differs from the chunking on disk. In general that's a good warning, but it shouldn't appear when the only difference between the on-disk chunking and the Dataset chunking is the last chunk.

What you expected to happen:

No warning.

Minimal Complete Verifiable Example:

```python

Create and write the data

import numpy as np import pandas as pd import xarray as xr

np.random.seed(0) temperature = 15 + 8 * np.random.randn(2, 2, 3) precipitation = 10 * np.random.rand(2, 2, 3) lon = [[-99.83, -99.32], [-99.79, -99.23]] lat = [[42.25, 42.21], [42.63, 42.59]] time = pd.date_range("2014-09-06", periods=3) reference_time = pd.Timestamp("2014-09-05") ds = xr.Dataset( data_vars=dict( temperature=(["x", "y", "time"], temperature), precipitation=(["x", "y", "time"], precipitation), ), coords=dict( lon=(["x", "y"], lon), lat=(["x", "y"], lat), time=time, reference_time=reference_time, ), attrs=dict(description="Weather related data."), ) ds2 = ds.chunk(chunks=dict(time=(2, 1))) ds2['temperature'].chunks

ds2.to_zarr("/tmp/test.zarr", mode="w") ```

Reading it produces a warning

python xr.open_zarr("/tmp/test.zarr") /mnt/c/Users/taugspurger/src/xarray/xarray/core/dataset.py:408: UserWarning: Specified Dask chunks (2, 1) would separate on disks chunk shape 2 for dimension time. This could degrade performance. Consider rechunking after loading instead. _check_chunks_compatibility(var, output_chunks, preferred_chunks)

Anything else we need to know?:

The check around https://github.com/pydata/xarray/blob/91318d2ee63149669404489be9198f230d877642/xarray/core/dataset.py#L371-L378 should probably ignore the very last chunk, since Zarr allows it to be different?

Environment:

Another attempt to fix #3698. The issue with my fix in is that we hit Variable._dask_finalize in both dask.optimize and dask.persist. We want to do the culling of unnecessary tasks (test_persist_Dataset) but only in the persist case, not optimize (test_optimize).

• [x] Closes #3698
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Previously we generated in invalidate Dask task graph, becuase the lines removed here dropped keys that were referenced elsewhere in the task graph. The original implementation had a comment indicating that this was to cull: https://github.com/pydata/xarray/blob/502a988ad5b87b9f3aeec3033bf55c71272e1053/xarray/core/variable.py#L384

Just spot-checking things, I think we're OK here though. Something like dask.visualize(arr[[0]], optimize_graph=True) indicates that we're OK.

• [x] Closes #3698
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

The examples on master raised with

pytb ValueError: Result from applying user function has unexpected coordinate variables {'month'}.

This PR updates the example to include the month coordinate. pytest --doctest-modules passes on these three now.

This reference an obj argument that only exists in parallel. The object being referenced is actually self.

[x] closes #3599

This fixes an issue with the HighLevelGraph noted in https://github.com/pydata/xarray/pull/3584, and exposed by a recent change in Dask to do more HLG fusion.

cc @dcherian.

Is xarray 3.5+ now? Congrats, I didn't realize that.

This started failing the benchmark machine, which I was tending to last night.

I'm doing a reduction like mean on a dask-backed DataArray, and passing it to dask.compute

```python In [3]: from dask import compute ...: import numpy as np ...: import xarray as xr ...:

In [4]: data = xr.DataArray(np.random.random(size=(10, 2)), ...: dims=['samples', 'features']).chunk((5, 2)) ...:

In [5]: compute(data.mean(axis=0)) ```

```pytb

ValueError Traceback (most recent call last) <ipython-input-5-47605102585c> in <module>() ----> 1 compute(data.mean(axis=0))

~/Envs/dask-dev/lib/python3.6/site-packages/dask/dask/base.py in compute(args, kwargs) 334 results_iter = iter(results) 335 return tuple(a if f is None else f(next(results_iter), a) --> 336 for f, a in postcomputes) 337 338

~/Envs/dask-dev/lib/python3.6/site-packages/dask/dask/base.py in <genexpr>(.0) 334 results_iter = iter(results) 335 return tuple(a if f is None else f(next(results_iter), *a) --> 336 for f, a in postcomputes) 337 338

~/Envs/dask-dev/lib/python3.6/site-packages/xarray/xarray/core/dataarray.py in _dask_finalize(results, func, args, name) 607 @staticmethod 608 def _dask_finalize(results, func, args, name): --> 609 ds = func(results, *args) 610 variable = ds._variables.pop(_THIS_ARRAY) 611 coords = ds._variables

~/Envs/dask-dev/lib/python3.6/site-packages/xarray/xarray/core/dataset.py in _dask_postcompute(results, info, args) 551 func, args2 = v 552 r = results2.pop() --> 553 result = func(r, args2) 554 else: 555 result = v

~/Envs/dask-dev/lib/python3.6/site-packages/xarray/xarray/core/variable.py in _dask_finalize(results, array_func, array_args, dims, attrs, encoding) 389 results = {k: v for k, v in results.items() if k[0] == name} # cull 390 data = array_func(results, *array_args) --> 391 return Variable(dims, data, attrs=attrs, encoding=encoding) 392 393 @property

~/Envs/dask-dev/lib/python3.6/site-packages/xarray/xarray/core/variable.py in init(self, dims, data, attrs, encoding, fastpath) 267 """ 268 self._data = as_compatible_data(data, fastpath=fastpath) --> 269 self._dims = self._parse_dimensions(dims) 270 self._attrs = None 271 self._encoding = None

~/Envs/dask-dev/lib/python3.6/site-packages/xarray/xarray/core/variable.py in _parse_dimensions(self, dims) 431 raise ValueError('dimensions %s must have the same length as the ' 432 'number of data dimensions, ndim=%s' --> 433 % (dims, self.ndim)) 434 return dims 435

ValueError: dimensions ('features',) must have the same length as the number of data dimensions, ndim=0 ```

The expected output is the .compute version, which works correctly:

python In [7]: data.mean(axis=0).compute() Out[7]: <xarray.DataArray (features: 2)> array([0.535643, 0.459406]) Dimensions without coordinates: features

Apologies if I'm doing something silly here, I don't know xarray :)

This should setup the proper links from the published output to the commit on Github.

FYI the benchmarks should be running stably now, and posted to http://pandas.pydata.org/speed/xarray. http://pandas.pydata.org/speed/xarray/regressions.xml has an RSS feed to the regressions.