issues

Is your feature request related to a problem?

We can do plain reductions, and groupby reductions with numbagg. Rolling is the last one left!

I don't think coarsen will benefit since it's basically a reshape and reduce on that view, so it should already be accelerated. There may be small gains in handling the boundary conditions but that's probably it.

Discussed in https://github.com/pydata/xarray/discussions/6901

Discussed in https://github.com/pydata/xarray/discussions/6762

What happened?

Really hard to describe in words =)

concat = xr.concat([da.isel(time=0), da.isel(time=[1])], dim="time") xr.align(concat, da, dim="time")

fails when concat and da should be identical. This is causing failures in cf-xarray:https://github.com/xarray-contrib/cf-xarray/issues/319

cc @benbovy

What did you expect to happen?

No response

Minimal Complete Verifiable Example

```Python import numpy as np import xarray as xr

time = xr.DataArray( np.array( ["2013-01-01T00:00:00.000000000", "2013-01-01T06:00:00.000000000"], dtype="datetime64[ns]", ), dims="time", name="time", )

da = time concat = xr.concat([da.isel(time=0), da.isel(time=[1])], dim="time") xr.align(da, concat, join="exact") # works

da = xr.DataArray(np.ones(time.shape), dims="time", coords={"time": time}) concat = xr.concat([da.isel(time=0), da.isel(time=[1])], dim="time") xr.align(da, concat, join="exact") ```

Relevant log output

```

ValueError Traceback (most recent call last) Input In [27], in <module> 17 da = xr.DataArray(np.ones(time.shape), dims="time", coords={"time": time}) 18 concat = xr.concat([da.isel(time=0), da.isel(time=[1])], dim="time") ---> 19 xr.align(da, concat, join="exact")

File ~/work/python/xarray/xarray/core/alignment.py:761, in align(join, copy, indexes, exclude, fill_value, *objects) 566 """ 567 Given any number of Dataset and/or DataArray objects, returns new 568 objects with aligned indexes and dimension sizes. (...) 751 752 """ 753 aligner = Aligner( 754 objects, 755 join=join, (...) 759 fill_value=fill_value, 760 ) --> 761 aligner.align() 762 return aligner.results

File ~/work/python/xarray/xarray/core/alignment.py:549, in Aligner.align(self) 547 self.find_matching_unindexed_dims() 548 self.assert_no_index_conflict() --> 549 self.align_indexes() 550 self.assert_unindexed_dim_sizes_equal() 552 if self.join == "override":

File ~/work/python/xarray/xarray/core/alignment.py:395, in Aligner.align_indexes(self) 393 if need_reindex: 394 if self.join == "exact": --> 395 raise ValueError( 396 "cannot align objects with join='exact' where " 397 "index/labels/sizes are not equal along " 398 "these coordinates (dimensions): " 399 + ", ".join(f"{name!r} {dims!r}" for name, dims in key[0]) 400 ) 401 joiner = self._get_index_joiner(index_cls) 402 joined_index = joiner(matching_indexes)

ValueError: cannot align objects with join='exact' where index/labels/sizes are not equal along these coordinates (dimensions): 'time' ('time',) ```

Anything else we need to know?

No response

Environment

xarray main

What is your issue?

The zarr backend has a list of "valid" encodings that needs to be updated any time zarr adds something new (e.g. https://github.com/pydata/xarray/pull/6348)

https://github.com/pydata/xarray/blob/53172cb1e03a98759faf77ef48efaa64676ad24a/xarray/backends/zarr.py#L215-L234

Can we get rid of this? I don't know the backends code well, but won't all our encoding parameters have been removed by this stage? The raise_on_invalid kwarg suggests so.

@tomwhite points out that zarr will raise a warning: ``` python

zarr.create((1,), blah=1) /Users/tom/miniconda/envs/sgkit-3.8/lib/python3.8/site-packages/zarr/creation.py:221: UserWarning: ignoring keyword argument 'blah' warn('ignoring keyword argument %r' % k) <zarr.core.Array (1,) float64> ```

Any thoughts on adding

1. .chunksize or .nbytes_chunk
2. .ntasks : this would be len(obj.__dask_graph__())

What happened:

The HTML repr is using the first element in a chunks tuple;

What you expected to happen:

it should be using whatever dask does in this case

Minimal Complete Verifiable Example:

```python

import xarray as xr import dask

test = xr.DataArray( dask.array.zeros( (12, 901, 1001), chunks=( (1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1), (1, 899, 1), (1, 199, 1, 199, 1, 199, 1, 199, 1, 199, 1), ), ) ) test.to_dataset(name="a") ```

EDIT: The text repr has the same issue <xarray.Dataset> Dimensions: (dim_0: 12, dim_1: 901, dim_2: 1001) Dimensions without coordinates: dim_0, dim_1, dim_2 Data variables: a (dim_0, dim_1, dim_2) float64 dask.array<chunksize=(1, 1, 1), meta=np.ndarray>

import matplotlib is failing when it imports PIL

```python

E ImportError: dlopen(/usr/local/miniconda/envs/xarray-tests/lib/python3.8/site-packages/PIL/_imaging.cpython-38-darwin.so, 2): Library not loaded: @rpath/libwebp.7.dylib E Referenced from: /usr/local/miniconda/envs/xarray-tests/lib/libtiff.5.dylib E Reason: Incompatible library version: libtiff.5.dylib requires version 9.0.0 or later, but libwebp.7.dylib provides version 8.0.0

/usr/local/miniconda/envs/xarray-tests/lib/python3.8/site-packages/PIL/Image.py:69: ImportError ```

MCVE Code Sample

Example 1

```python import xarray as xr import numpy as np

da = xr.DataArray(np.random.randn(10, 20), name="a") ds = da.to_dataset() xr.testing.assert_equal(da, da.chunk({"dim_0": 2})) # works xr.testing.assert_equal(da.chunk(), da.chunk({"dim_0": 2})) # works

xr.testing.assert_equal(ds, ds.chunk({"dim_0": 2})) # works xr.testing.assert_equal(ds.chunk(), ds.chunk({"dim_0": 2})) # does not work

``` I get

```

ValueError Traceback (most recent call last) <ipython-input-1-bc8216a67408> in <module> 8 9 xr.testing.assert_equal(ds, ds.chunk({"dim_0": 2})) # works ---> 10 xr.testing.assert_equal(ds.chunk(), ds.chunk({"dim_0": 2})) # does not work

~/work/python/xarray/xarray/testing.py in assert_equal(a, b) 56 assert a.equals(b), formatting.diff_array_repr(a, b, "equals") 57 elif isinstance(a, Dataset): ---> 58 assert a.equals(b), formatting.diff_dataset_repr(a, b, "equals") 59 else: 60 raise TypeError("{} not supported by assertion comparison".format(type(a)))

~/work/python/xarray/xarray/core/dataset.py in equals(self, other) 1322 """ 1323 try: -> 1324 return self._all_compat(other, "equals") 1325 except (TypeError, AttributeError): 1326 return False

~/work/python/xarray/xarray/core/dataset.py in _all_compat(self, other, compat_str) 1285 1286 return self._coord_names == other._coord_names and utils.dict_equiv( -> 1287 self._variables, other._variables, compat=compat 1288 ) 1289

~/work/python/xarray/xarray/core/utils.py in dict_equiv(first, second, compat) 335 """ 336 for k in first: --> 337 if k not in second or not compat(first[k], second[k]): 338 return False 339 for k in second:

~/work/python/xarray/xarray/core/dataset.py in compat(x, y) 1282 # require matching order for equality 1283 def compat(x: Variable, y: Variable) -> bool: -> 1284 return getattr(x, compat_str)(y) 1285 1286 return self._coord_names == other._coord_names and utils.dict_equiv(

~/work/python/xarray/xarray/core/variable.py in equals(self, other, equiv) 1558 try: 1559 return self.dims == other.dims and ( -> 1560 self._data is other._data or equiv(self.data, other.data) 1561 ) 1562 except (TypeError, AttributeError):

~/work/python/xarray/xarray/core/duck_array_ops.py in array_equiv(arr1, arr2) 201 warnings.filterwarnings("ignore", "In the future, 'NAT == x'") 202 flag_array = (arr1 == arr2) | (isnull(arr1) & isnull(arr2)) --> 203 return bool(flag_array.all()) 204 205

~/miniconda3/envs/dcpy/lib/python3.7/site-packages/dask/array/core.py in bool(self) 1380 ) 1381 else: -> 1382 return bool(self.compute()) 1383 1384 nonzero = bool # python 2

~/miniconda3/envs/dcpy/lib/python3.7/site-packages/dask/base.py in compute(self, kwargs) 173 dask.base.compute 174 """ --> 175 (result,) = compute(self, traverse=False, kwargs) 176 return result 177

~/miniconda3/envs/dcpy/lib/python3.7/site-packages/dask/base.py in compute(args, kwargs) 444 keys = [x.dask_keys() for x in collections] 445 postcomputes = [x.dask_postcompute() for x in collections] --> 446 results = schedule(dsk, keys, kwargs) 447 return repack([f(r, a) for r, (f, a) in zip(results, postcomputes)]) 448

~/miniconda3/envs/dcpy/lib/python3.7/site-packages/dask/threaded.py in get(dsk, result, cache, num_workers, pool, kwargs) 80 get_id=_thread_get_id, 81 pack_exception=pack_exception, ---> 82 kwargs 83 ) 84

~/miniconda3/envs/dcpy/lib/python3.7/site-packages/dask/local.py in get_async(apply_async, num_workers, dsk, result, cache, get_id, rerun_exceptions_locally, pack_exception, raise_exception, callbacks, dumps, loads, **kwargs) 489 _execute_task(task, data) # Re-execute locally 490 else: --> 491 raise_exception(exc, tb) 492 res, worker_id = loads(res_info) 493 state["cache"][key] = res

~/miniconda3/envs/dcpy/lib/python3.7/site-packages/dask/compatibility.py in reraise(exc, tb) 128 if exc.traceback is not tb: 129 raise exc.with_traceback(tb) --> 130 raise exc 131 132 import pickle as cPickle

~/miniconda3/envs/dcpy/lib/python3.7/site-packages/dask/local.py in execute_task(key, task_info, dumps, loads, get_id, pack_exception) 231 try: 232 task, data = loads(task_info) --> 233 result = _execute_task(task, data) 234 id = get_id() 235 result = dumps((result, id))

~/miniconda3/envs/dcpy/lib/python3.7/site-packages/dask/core.py in _execute_task(arg, cache, dsk) 117 func, args = arg[0], arg[1:] 118 args2 = [_execute_task(a, cache) for a in args] --> 119 return func(*args2) 120 elif not ishashable(arg): 121 return arg

~/miniconda3/envs/dcpy/lib/python3.7/site-packages/dask/optimization.py in call(self, *args) 1057 if not len(args) == len(self.inkeys): 1058 raise ValueError("Expected %d args, got %d" % (len(self.inkeys), len(args))) -> 1059 return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args))) 1060 1061 def reduce(self):

~/miniconda3/envs/dcpy/lib/python3.7/site-packages/dask/core.py in get(dsk, out, cache) 147 for key in toposort(dsk): 148 task = dsk[key] --> 149 result = _execute_task(task, cache) 150 cache[key] = result 151 result = _execute_task(out, cache)

~/miniconda3/envs/dcpy/lib/python3.7/site-packages/dask/core.py in _execute_task(arg, cache, dsk) 117 func, args = arg[0], arg[1:] 118 args2 = [_execute_task(a, cache) for a in args] --> 119 return func(*args2) 120 elif not ishashable(arg): 121 return arg

ValueError: operands could not be broadcast together with shapes (0,20) (2,20) ```

Example 2

The relevant xarray line in the previous traceback is flag_array = (arr1 == arr2) | (isnull(arr1) & isnull(arr2)), so I tried

python (ds.isnull() & ds.chunk({"dim_0": 1}).isnull()).compute() # works (ds.chunk().isnull() & ds.chunk({"dim_0": 1}).isnull()).compute() # does not work?!

```

ValueError Traceback (most recent call last) <ipython-input-4-abdfbeda355a> in <module> 1 (ds.isnull() & ds.chunk({"dim_0": 1}).isnull()).compute() # works ----> 2 (ds.chunk().isnull() & ds.chunk({"dim_0": 1}).isnull()).compute() # does not work?!

~/work/python/xarray/xarray/core/dataset.py in compute(self, kwargs) 791 """ 792 new = self.copy(deep=False) --> 793 return new.load(kwargs) 794 795 def _persist_inplace(self, **kwargs) -> "Dataset":

~/work/python/xarray/xarray/core/dataset.py in load(self, **kwargs) 645 646 for k, data in zip(lazy_data, evaluated_data): --> 647 self.variables[k].data = data 648 649 # load everything else sequentially

~/work/python/xarray/xarray/core/variable.py in data(self, data) 331 data = as_compatible_data(data) 332 if data.shape != self.shape: --> 333 raise ValueError("replacement data must match the Variable's shape") 334 self._data = data 335

ValueError: replacement data must match the Variable's shape ```

Problem Description

I don't know what's going on here. I expect assert_equal should return True for all these examples.

Our test for isnull with dask always calls load before comparing:

def test_isnull_with_dask(): da = construct_dataarray(2, np.float32, contains_nan=True, dask=True) assert isinstance(da.isnull().data, dask_array_type) assert_equal(da.isnull().load(), da.load().isnull())

Output of xr.show_versions()

xarray master & dask 2.3.0

Dask has updated it's percentile, quantile implementation: https://github.com/dask/dask/pull/4677

Can we now update our median method to work with dask arrays?

```python =================================== FAILURES =================================== ____ TestVariable.test_index_0d_not_a_time _______ self = <xarray.tests.test_variable.TestVariable object at 0x7f0dd7b6bda0> def test_index_0d_not_a_time(self): d = np.datetime64('NaT', 'ns') x = self.cls(['x'], [d])

  self._assertIndexedLikeNDArray(x, d)

xarray/tests/test_variable.py:197:

self = <xarray.tests.test_variable.TestVariable object at 0x7f0dd7b6bda0> variable = <xarray.Variable (x: 1)> array(['NaT'], dtype='datetime64[ns]') expected_value0 = numpy.datetime64('NaT'), expected_dtype = None def _assertIndexedLikeNDArray(self, variable, expected_value0, expected_dtype=None): """Given a 1-dimensional variable, verify that the variable is indexed like a numpy.ndarray. """ assert variable[0].shape == () assert variable[0].ndim == 0 assert variable[0].size == 1 # test identity assert variable.equals(variable.copy()) assert variable.identical(variable.copy()) # check value is equal for both ndarray and Variable with warnings.catch_warnings(): warnings.filterwarnings('ignore', "In the future, 'NAT == x'")

      assert variable.values[0] == expected_value0

E AssertionError: assert numpy.datetime64('NaT') == numpy.datetime64('NaT') E -numpy.datetime64('NaT') E +numpy.datetime64('NaT') xarray/tests/test_variable.py:143: AssertionError

____ TestVariableWithDask.test_index_0d_not_a_time _____ self = <xarray.tests.test_variable.TestVariableWithDask object at 0x7f0e00bc5978> def test_index_0d_not_a_time(self): d = np.datetime64('NaT', 'ns') x = self.cls(['x'], [d])

  self._assertIndexedLikeNDArray(x, d)

xarray/tests/test_variable.py:197:

self = <xarray.tests.test_variable.TestVariableWithDask object at 0x7f0e00bc5978> variable = <xarray.Variable (x: 1)> dask.array<shape=(1,), dtype=datetime64[ns], chunksize=(1,)> expected_value0 = numpy.datetime64('NaT'), expected_dtype = None def _assertIndexedLikeNDArray(self, variable, expected_value0, expected_dtype=None): """Given a 1-dimensional variable, verify that the variable is indexed like a numpy.ndarray. """ assert variable[0].shape == () assert variable[0].ndim == 0 assert variable[0].size == 1 # test identity assert variable.equals(variable.copy()) assert variable.identical(variable.copy()) # check value is equal for both ndarray and Variable with warnings.catch_warnings(): warnings.filterwarnings('ignore', "In the future, 'NAT == x'")

      assert variable.values[0] == expected_value0

E AssertionError: assert numpy.datetime64('NaT') == numpy.datetime64('NaT') E -numpy.datetime64('NaT') E +numpy.datetime64('NaT') xarray/tests/test_variable.py:143: AssertionError

___ TestIndexVariable.testindex_0d_not_a_time ____ self = <xarray.tests.test_variable.TestIndexVariable object at 0x7f0e01063390> def test_index_0d_not_a_time(self): d = np.datetime64('NaT', 'ns') x = self.cls(['x'], [d])

  self._assertIndexedLikeNDArray(x, d)

xarray/tests/test_variable.py:197:

self = <xarray.tests.test_variable.TestIndexVariable object at 0x7f0e01063390> variable = <xarray.IndexVariable 'x' (x: 1)> array(['NaT'], dtype='datetime64[ns]') expected_value0 = numpy.datetime64('NaT'), expected_dtype = None def _assertIndexedLikeNDArray(self, variable, expected_value0, expected_dtype=None): """Given a 1-dimensional variable, verify that the variable is indexed like a numpy.ndarray. """ assert variable[0].shape == () assert variable[0].ndim == 0 assert variable[0].size == 1 # test identity assert variable.equals(variable.copy()) assert variable.identical(variable.copy()) # check value is equal for both ndarray and Variable with warnings.catch_warnings(): warnings.filterwarnings('ignore', "In the future, 'NAT == x'")

      assert variable.values[0] == expected_value0

E AssertionError: assert numpy.datetime64('NaT') == numpy.datetime64('NaT') E -numpy.datetime64('NaT') E +numpy.datetime64('NaT') xarray/tests/test_variable.py:143: AssertionError ```

It would be useful to add the boolean kwargs logx, logy, loglog to DataArray.plot() just as in pandas.DataFrame.plot()

https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.plot.html#pandas.DataFrame.plot