What is your issue?

cc @hmaarrfk @kmuehlbauer

Passed: https://github.com/pydata/xarray/actions/runs/4115923717/jobs/7105298426 Failed: https://github.com/pydata/xarray/actions/runs/4115946392/jobs/7105345290

Versions: h5netcdf 1.1.0 pyhd8ed1ab_0 conda-forge h5py 3.8.0 nompi_py310h5555e59_100 conda-forge hdf4 4.2.15 h7aa5921_5 conda-forge hdf5 1.12.2 nompi_h48135f9_101 conda-forge

``` =================================== FAILURES =================================== ___ test_open_mfdataset_manyfiles[h5netcdf-20-True-5-5] ______ [gw1] darwin -- Python 3.10.9 /Users/runner/micromamba-root/envs/xarray-tests/bin/python

readengine = 'h5netcdf', nfiles = 20, parallel = True, chunks = 5 file_cache_maxsize = 5

@requires_dask
@pytest.mark.filterwarnings("ignore:use make_scale(name) instead")
def test_open_mfdataset_manyfiles(
    readengine, nfiles, parallel, chunks, file_cache_maxsize
):
    # skip certain combinations
    skip_if_not_engine(readengine)

    if ON_WINDOWS:
        pytest.skip("Skipping on Windows")

    randdata = np.random.randn(nfiles)
    original = Dataset({"foo": ("x", randdata)})
    # test standard open_mfdataset approach with too many files
    with create_tmp_files(nfiles) as tmpfiles:
        writeengine = readengine if readengine != "pynio" else "netcdf4"
        # split into multiple sets of temp files
        for ii in original.x.values:
            subds = original.isel(x=slice(ii, ii + 1))
            if writeengine != "zarr":
                subds.to_netcdf(tmpfiles[ii], engine=writeengine)
            else:  # if writeengine == "zarr":
                subds.to_zarr(store=tmpfiles[ii])

        # check that calculation on opened datasets works properly

      with open_mfdataset(
            tmpfiles,
            combine="nested",
            concat_dim="x",
            engine=readengine,
            parallel=parallel,
            chunks=chunks if (not chunks and readengine != "zarr") else "auto",
        ) as actual:

/Users/runner/work/xarray/xarray/xarray/tests/test_backends.py:3267:

/Users/runner/work/xarray/xarray/xarray/backends/api.py:991: in open_mfdataset datasets, closers = dask.compute(datasets, closers) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/base.py:599: in compute results = schedule(dsk, keys, kwargs) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/threaded.py:89: in get results = get_async( /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py:511: in get_async raise_exception(exc, tb) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py:319: in reraise raise exc /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py:224: in execute_task result = _execute_task(task, data) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/core.py:119: in _execute_task return func((_execute_task(a, cache) for a in args)) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/utils.py:72: in apply return func(args, kwargs) /Users/runner/work/xarray/xarray/xarray/backends/api.py:526: in open_dataset backend_ds = backend.open_dataset( /Users/runner/work/xarray/xarray/xarray/backends/h5netcdf_.py:417: in open_dataset ds = store_entrypoint.open_dataset( /Users/runner/work/xarray/xarray/xarray/backends/store.py:32: in open_dataset vars, attrs = store.load() /Users/runner/work/xarray/xarray/xarray/backends/common.py:129: in load (decode_variable_name(k), v) for k, v in self.get_variables().items() /Users/runner/work/xarray/xarray/xarray/backends/h5netcdf.py:220: in get_variables return FrozenDict( /Users/runner/work/xarray/xarray/xarray/core/utils.py:471: in FrozenDict return Frozen(dict(args, *kwargs)) /Users/runner/work/xarray/xarray/xarray/backends/h5netcdf_.py:221: in <genexpr> (k, self.open_store_variable(k, v)) for k, v in self.ds.variables.items() /Users/runner/work/xarray/xarray/xarray/backends/h5netcdf_.py:200: in open_store_variable elif var.compression is not None: /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/h5netcdf/core.py:394: in compression return self._h5ds.compression

self = <[AttributeError("'NoneType' object has no attribute '_root'") raised in repr()] Variable object at 0x151378970>

@property
def _h5ds(self):
    # Always refer to the root file and store not h5py object
    # subclasses:

  return self._root._h5file[self._h5path]

E AttributeError: 'NoneType' object has no attribute '_h5file'

```

Discussed in https://github.com/pydata/xarray/discussions/8751

What happened?

Currently we don't overwrite indexes when region="auto" https://github.com/pydata/xarray/blob/e6ccedb56ed4bc8d0b7c1f16ab325795330fb19a/xarray/backends/api.py#L1769-L1770

I propose we do this for all region writes and completely disallow modifying indexes with a region write.

This would match the map_blocks model, where all indexes are specified in the template and no changes by the mapped function are allowed.

What is your issue?

This line fails with pd.set_options("mode.copy_on_write", True) https://github.com/pydata/xarray/blob/c9d3084e98d38a7a9488380789a8d0acfde3256f/xarray/tests/init.py#L329

We'll need to fix this before Pandas 3 is released in April: https://github.com/pydata/xarray/blob/c9d3084e98d38a7a9488380789a8d0acfde3256f/xarray/tests/init.py#L329

Here's a test ```python def example(): obj = Dataset() obj["dim2"] = ("dim2", 0.5 * np.arange(9)) obj["time"] = ("time", pd.date_range("2000-01-01", periods=20) print({k: v.data.flags for k, v in obj.variables.items()}) return obj

example() pd.set_options("mode.copy_on_write", True) example() ```

What happened?

(xref #8658)

I'm not sure what the expected behaviour is here:

```python import xarray as xr import numpy as np from xarray.testing import _assert_internal_invariants

ds = xr.Dataset() ds.coords["1"] = ("1", np.array([1], dtype=np.uint32)) ds["1_"] = ("1", np.array([1], dtype=np.uint32)) ds = ds.rename_vars({"1": "0"}) ds ```

It looks like this sequence of operations creates a default index

But then ```python from xarray.testing import _assert_internal_invariants

_assert_internal_invariants(ds, check_default_indexes=True) fails with ... File ~/repos/xarray/xarray/testing/assertions.py:301, in _assert_indexes_invariants_checks(indexes, possible_coord_variables, dims, check_default) 299 if check_default: 300 defaults = default_indexes(possible_coord_variables, dims) --> 301 assert indexes.keys() == defaults.keys(), (set(indexes), set(defaults)) 302 assert all(v.equals(defaults[k]) for k, v in indexes.items()), ( 303 indexes, 304 defaults, 305 )

AssertionError: ({'0'}, set()) ```

What happened?

All tests are failing because of an error in create_test_data

from xarray.tests import create_test_data create_test_data()

```

AssertionError Traceback (most recent call last) Cell In[3], line 2 1 from xarray.tests import create_test_data ----> 2 create_test_data()

File ~/repos/xarray/xarray/tests/init.py:329, in create_test_data(seed, add_attrs, dim_sizes) 327 obj.coords["numbers"] = ("dim3", numbers_values) 328 obj.encoding = {"foo": "bar"} --> 329 assert all(var.values.flags.writeable for var in obj.variables.values()) 330 return obj

AssertionError: ```

Is your feature request related to a problem?

If we have two objects with dims x and x1, then xr.align(..., join="exact") will pass because these dimensions are broadcastable.

I'd like a stricter option (join="strict"?) that disallows broadcasting.

Describe the solution you'd like

python xr.align( xr.DataArray([1], dims="x"), xr.DataArray([1], dims="x1"), join="strict", ) would raise an error.

It'd be nice to have this as a built-in option so we can use python with xr.set_options(arithmetic_join="strict"): ...

Describe alternatives you've considered

An alternative would be to allow control over automatic broadcasting through the set_options context manager., but that seems like it would be more complicated to implement.

Additional context

This turns up in staggered grid calculations with xgcm where it is easy to mistakenly construct very high-dimensional arrays because of automatic broadcasting.

What is your issue?

Thanks to @keewis for volunteering at today's meeting :()

What is your issue?

We haven't done this yet! https://github.com/pydata/xarray/blob/d87ba61c957fc3af77251ca6db0f6bccca1acb82/pyproject.toml#L11-L15

What is your issue?

Thanks @TomNicholas for volunteering to drive this release!

What is your issue?

Looks like we can bump a number of min versions: ``` Package Required Policy Status

cartopy 0.20 (2021-09-17) 0.21 (2022-09-10) < dask-core 2022.7 (2022-07-08) 2022.12 (2022-12-02) < distributed 2022.7 (2022-07-08) 2022.12 (2022-12-02) < flox 0.5 (2022-05-03) 0.6 (2022-10-12) < iris 3.2 (2022-02-15) 3.4 (2022-12-01) < matplotlib-base 3.5 (2021-11-18) 3.6 (2022-09-16) < numba 0.55 (2022-01-14) 0.56 (2022-09-28) < numpy 1.22 (2022-01-03) 1.23 (2022-06-23) < packaging 21.3 (2021-11-18) 22.0 (2022-12-08) < pandas 1.4 (2022-01-22) 1.5 (2022-09-19) < scipy 1.8 (2022-02-06) 1.9 (2022-07-30) < seaborn 0.11 (2020-09-08) 0.12 (2022-09-06) < typing_extensions 4.3 (2022-07-01) 4.4 (2022-10-07) < zarr 2.12 (2022-06-23) 2.13 (2022-09-27) < ```

What happened?

xarray/namedarray/core.py:758: error: Value of type Never is not indexable [index] xarray/core/alignment.py:684: error: Unused "type: ignore" comment [unused-ignore] xarray/core/alignment.py:1156: error: Unused "type: ignore" comment [unused-ignore] xarray/core/dataset.py: note: In member "sortby" of class "Dataset": xarray/core/dataset.py:7967: error: Incompatible types in assignment (expression has type "tuple[Alignable, ...]", variable has type "tuple[DataArray, ...]") [assignment] xarray/core/dataset.py:7979: error: "Alignable" has no attribute "isel" [attr-defined]

Is your feature request related to a problem?

We should document changing the time vector using pandas date offsets here

This is particularly useful for centering the time stamps after a resampling operation.

Related: - CFTime offsets: https://github.com/pydata/xarray/issues/5687 - loffset deprecation: https://github.com/pydata/xarray/pull/7444

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Is your feature request related to a problem?

Shall we support opt_einsum as an optional backend for xr.dot?

opt_einsum.contract is a drop-in replacement for np.einsum so this monkey-patch works today xr.core.duck_array_ops.einsum = opt_einsum.contract

Describe the solution you'd like

Add a backend kwarg with options "numpy" and "opt_einsum", with the default being "numpy"

Describe alternatives you've considered

We could create a new package but it seems a bit silly.

Additional context

No response

We've accumulated a nice number of changes.

Can someone volunteer to do a release in the next few days?

Is your feature request related to a problem?

Consider this example

python ds = xr.Dataset({"a": ("x", [1, 2, 3]), "b": ("y", [1, 2, 3])}) xr.apply_ufunc(np.mean, ds, input_core_dims=[["x"]])

This raises ValueError: operand to apply_ufunc has required core dimensions ['x'], but some of these dimensions are absent on an input variable: ['x']

because core dimension x is missing on variable b. This behaviour makes it annoying to use apply_ufunc on Datasets.

Describe the solution you'd like

Add a new kwarg to apply_ufunc called missing_core_dim that controls how to handle variables without all input core dimensions. This kwarg could take one of two values: 1. "raise" - raise an error, current behaviour 2. "copy" - skip applying the function and copy the variable from input to output. 3. "drop"- skip applying the function and drop the variable.

Describe alternatives you've considered

No response

Additional context

No response

What happened?

See https://github.com/pydata/xarray/pull/8101

FAILED xarray/tests/test_missing.py::test_interpolate_pd_compat - ValueError: 'fill_value' is not a valid keyword for DataFrame.interpolate FAILED xarray/tests/test_missing.py::test_interpolate_pd_compat_non_uniform_index - ValueError: 'fill_value' is not a valid keyword for DataFrame.interpolate

and this doctest FAILED xarray/core/dataarray.py::xarray.core.dataarray.DataArray.to_unstacked_dataset @pydata/xarray can someone take a look please?

Should say that Y and M cannot be supported with datetime64

Discussed in https://github.com/pydata/xarray/discussions/8000

Is your feature request related to a problem?

I opened this PR to add minimum versions for our optional dependencies: https://github.com/conda-forge/xarray-feedstock/pull/84/files to prevent issues like #7467

I think we'd need a policy to choose which ones to list. Here's the current list: run_constrained: - bottleneck >=1.3 - cartopy >=0.20 - cftime >=1.5 - dask-core >=2022.1 - distributed >=2022.1 - flox >=0.5 - h5netcdf >=0.13 - h5py >=3.6 - hdf5 >=1.12 - iris >=3.1 - matplotlib-base >=3.5 - nc-time-axis >=1.4 - netcdf4 >=1.5.7 - numba >=0.55 - pint >=0.18 - scipy >=1.7 - seaborn >=0.11 - sparse >=0.13 - toolz >=0.11 - zarr >=2.10

Some examples to think about: 1. iris seems like a bad one to force. It seems like people might use Iris and Xarray independently and Xarray shouldn't force a minimum version. 2. For backends, I arbitrarily kept netcdf4, h5netcdf and zarr. 3. It seems like we should keep array types: so dask, sparse, pint.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

What happened?

From https://discourse.pangeo.io/t/xarray-unable-to-allocate-memory-how-to-size-up-problem/3233/4

Consider ``` python

ds is dataset with big dask arrays

mean = ds.groupby("time.day").mean() mean.to_netcdf() mean = xr.open_dataset(...)

ds.groupby("time.day") - mean ```

In GroupBy._binary_op https://github.com/pydata/xarray/blob/39caafae4452f5327a7cd671b18d4bb3eb3785ba/xarray/core/groupby.py#L616

we will eagerly construct other that is of the same size as ds.

What did you expect to happen?

I think the only solution is to automatically chunk if ds has dask arrays, and other (or mean) isn't backed by dask arrays. A chunk size of 1 seems sensible.

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

What happened?

I just ran in to this error in an environment without dask. TypeError: Could not find a Chunk Manager which recognises type <class 'dask.array.core.Array'>

I think we could easily recommend the user to install a package that provides dask by looking at type(array).__name__. This would make the message a lot friendlier

What is your issue?

See https://github.com/matplotlib/matplotlib/issues/16296

Looks like the MPL API may have advanced enough that we can delete some of our use of private attributes.

What happened?

```python with xr.set_options(display_style="text", use_flox=False): with xr.set_options(use_flox=False): display( xr.DataArray( data=np.array([np.nan, 1, 1, np.nan, 1, 1]), dims="x", coords={"labels": ("x", np.array([1, 2, 3, 1, 2, 3]))}, ) .groupby("labels") .sum() )

with xr.set_options(use_flox=True):
    display(
        xr.DataArray(
            data=np.array([np.nan, 1, 1, np.nan, 1, 1]),
            dims="x",
            coords={"labels": ("x", np.array([1, 2, 3, 1, 2, 3]))},
        )
        .groupby("labels")
        .sum()
    )

```

```

without flox

<xarray.DataArray (labels: 3)> array([0., 2., 2.]) Coordinates: * labels (labels) int64 1 2 3

with flox

<xarray.DataArray (labels: 3)> array([nan, 2., 2.]) Coordinates: * labels (labels) int64 1 2 3 ```

What did you expect to happen?

The same answer. We should set min_count=0 when min_count is None

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

What happened?

Pandas 2 is out.

We have a pandas<2 pin for our latest release, but mamba is now returning xarray=2023.1.0 and pandas=2.0 which is making cf-xarray and flox tests fail.

It looks like any project that tests resample without pinning pandas will fail.

I opened the issue here for visibility. It seems we might need a repodata patch to disallow pandas<2?

cc @ocefpaf

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Is your feature request related to a problem?

Given all the outreachy interest (and perhaps just in general) it may be nice to enable a welcome bot like on the Jupyter repos

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

What happened?

MY prototype CRSIndex is broken on the release version: https://github.com/dcherian/crsindex/blob/main/crsindex.ipynb under heading "BROKEN: Successfully align with a copy of itself"

The cell's code is : copy = newds.copy(deep=True) xr.align(copy, newds)

which should always work.

@headtr1ck is https://github.com/pydata/xarray/pull/7140 to blame?

Environment

Minimal Complete Verifiable Example:

Here's a small array with too-small chunk sizes just as an example ```python

Put your MCVE code here

import dask.array import xarray as xr

var = xr.Variable(("x", "y", "z"), dask.array.random.random((4, 18483, 1000), chunks=(1, 183, -1))) ```

Now stack two dimensions, this is a 100x increase in chunk size (in my actual code, 85MB chunks become 8.5GB chunks =) )

var.stack(new=("x", "y"))

But calling reshape on the dask array preserves the original chunk size var.data.reshape((4*18483, -1))

Solution

Ah, found it , we transpose then reshape in Variable_stack_once. https://github.com/pydata/xarray/blob/f915515d610b4471888fa44dfb00dbae3fd22349/xarray/core/variable.py#L1521-L1527

Writing those steps with pure dask yields the same 100x increase in chunksize

python var.data.transpose([2, 0, 1]).reshape((-1, 4*18483))

Anything else we need to know?:

Environment:

What happened?

logs sphinx.errors.SphinxParallelError: RuntimeError: Non Expected exception in `/home/docs/checkouts/readthedocs.org/user_builds/xray/checkouts/7433/doc/user-guide/interpolation.rst` line 331

This seems real

What happened?

For "small" datasets, we load in to memory when displaying the repr. For cloud backed datasets with large number of "small" variables, this can use a lot of time sequentially loading O(100) variables just for a repr.

https://github.com/pydata/xarray/blob/6c8db5ed005e000b35ad8b6ea9080105e608e976/xarray/core/formatting.py#L548-L549

What did you expect to happen?

Fast reprs!

Minimal Complete Verifiable Example

This dataset has 48 "small" variables ```Python import xarray as xr

dc1 = xr.open_dataset('s3://its-live-data/datacubes/v02/N40E080/ITS_LIVE_vel_EPSG32645_G0120_X250000_Y4750000.zarr', engine= 'zarr', storage_options = {'anon':True}) dc1.repr_html() ```

On 2022.03.0 this repr takes 36.4s If I comment the array.size condition I get 6μs.

MVCE confirmation

• [x] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [x] Complete example — the example is self-contained, including all data and the text of any traceback.
• [x] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [x] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Is your feature request related to a problem?

No.

Describe the solution you'd like

If possible, I think it'd be nice to move a lot of the upstream-dev CI scripting to its own github action like "ci-trigger". This will make it easier to use in other projects (like those under xarray-contrib). I'd like to use it for flox, cf-xarray.

Describe alternatives you've considered

No response

Additional context

No response

What happened?

From upstream-dev CI: Workflow Run URL

Discussed in https://github.com/pydata/xarray/discussions/6901

What is your issue?

It's been 3 months since our last release.

We still have quite a few regressions from the last release but @benbovy does have open PRs for a number of them. However, we do have some nice bugfixes and other commits in the mean time.

I propose we issue a new release, perhaps after @benbovy merges the PRs he thinks are ready.

I'll be out of town for the next few days, so if someone else could volunteer to be release manager that would be great!

Seen in #4103

MCVE Code Sample

```python from xarray.tests import assert_identical

coord_1 = xr.DataArray([1, 2], dims=["coord_1"], attrs={"attrs": True}) da = xr.DataArray([1, 0], [coord_1]) obj = da.reset_index("coord_1").rename({"coord_1_": "coord_1"}) assert_identical(da, obj) ```

Expected Output

Problem Description

``` AssertionErrorTraceback (most recent call last) <ipython-input-19-02ef6bd89884> in <module> ----> 1 assert_identical(da, obj)

~/work/python/xarray/xarray/tests/init.py in assert_identical(a, b) 160 xarray.testing.assert_identical(a, b) 161 xarray.testing._assert_internal_invariants(a) --> 162 xarray.testing._assert_internal_invariants(b) 163 164

~/work/python/xarray/xarray/testing.py in _assert_internal_invariants(xarray_obj) 265 _assert_variable_invariants(xarray_obj) 266 elif isinstance(xarray_obj, DataArray): --> 267 _assert_dataarray_invariants(xarray_obj) 268 elif isinstance(xarray_obj, Dataset): 269 _assert_dataset_invariants(xarray_obj)

~/work/python/xarray/xarray/testing.py in _assert_dataarray_invariants(da) 210 assert all( 211 isinstance(v, IndexVariable) for (k, v) in da._coords.items() if v.dims == (k,) --> 212 ), {k: type(v) for k, v in da._coords.items()} 213 for k, v in da._coords.items(): 214 _assert_variable_invariants(v, k)

AssertionError: {'coord_1': <class 'xarray.core.variable.Variable'>} ```

Versions

What happened?

Seen in #4972

``` python import xarray as xr import numpy as np

array = np.array([0. +0.j, 0.+np.nan * 1j], dtype=np.complex64) var = xr.Variable("x", array) print(var.mean().data) print(array.mean()) ```

0j (nan+nanj)

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Discussed in https://github.com/pydata/xarray/discussions/6762

Discussed in https://github.com/pydata/xarray/discussions/6738

What happened?

Seen over in cf_xarray

Using xr.core.resample.Resample worked prior to https://github.com/pydata/xarray/pull/6702.

Now we need to use from xarray.core.resample import Resample

I don't know if this is something that needs to be fixed or only worked coincidentally earlier. But I thought it was worth discussing prior to release.

Thanks to @aulemahal for spotting

I think we only describe available options in the API reference for xr.set_options It'd be nice to add a "Configuring Xarray" section in the User Guide.

What is your issue?

@aurghs developed some nice backend tutorial material for the Dask Summit: https://github.com/aurghs/xarray-backend-tutorial

It'd be nice to add it either to our main documentation or to https://github.com/xarray-contrib/xarray-tutorial.

Is your feature request related to a problem?

From @keewis in #6645

if anyone can figure out how to create PEP440 (and thus PyPI) compatible development versions I think we can have a CI publish every commit on main to TestPyPI.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

What happened?

From @malmans2

``` python import numpy as np

import xarray as xr

ds = xr.Dataset( dict(a=(("z",), np.ones(10))), coords=dict(b=(("z"), np.arange(2).repeat(5)), c=(("z"), np.arange(5).repeat(2))), ).set_index(bc=["b", "c"]) grouped = ds.groupby("bc")

with xr.set_options(use_flox=False): grouped.sum() # OK

with xr.set_options(use_flox=True): grouped.sum() # Error ```

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python ctests/test_xarray.py:329: in test_multi_index_groupby_sum actual = xarray_reduce(ds, "bc", func="sum") flox/xarray.py:374: in xarray_reduce actual[k] = v.expand_dims(missing_group_dims) ../xarray/xarray/core/dataset.py:1427: in setitem self.update({key: value}) ../xarray/xarray/core/dataset.py:4432: in update merge_result = dataset_update_method(self, other) ../xarray/xarray/core/merge.py:1070: in dataset_update_method return merge_core( ../xarray/xarray/core/merge.py:722: in merge_core aligned = deep_align( ../xarray/xarray/core/alignment.py:824: in deep_align aligned = align( ../xarray/xarray/core/alignment.py:761: in align aligner.align() ../xarray/xarray/core/alignment.py:550: in align self.assert_unindexed_dim_sizes_equal() ../xarray/xarray/core/alignment.py:450: in assert_unindexed_dim_sizes_equal raise ValueError( E ValueError: cannot reindex or align along dimension 'bc' because of conflicting dimension sizes: {10, 6} (note: an index is found along that dimension with size=10) ____ test_multi_index_groupby_sum[numpy] _______________________________ tests/test_xarray.py:329: in test_multi_index_groupby_sum actual = xarray_reduce(ds, "bc", func="sum") flox/xarray.py:374: in xarray_reduce actual[k] = v.expand_dims(missing_group_dims) ../xarray/xarray/core/dataset.py:1427: in __setitem self.update({key: value}) ../xarray/xarray/core/dataset.py:4432: in update merge_result = dataset_update_method(self, other) ../xarray/xarray/core/merge.py:1070: in dataset_update_method return merge_core( ../xarray/xarray/core/merge.py:722: in merge_core aligned = deep_align( ../xarray/xarray/core/alignment.py:824: in deep_align aligned = align( ../xarray/xarray/core/alignment.py:761: in align aligner.align() ../xarray/xarray/core/alignment.py:550: in align self.assert_unindexed_dim_sizes_equal() ../xarray/xarray/core/alignment.py:450: in assert_unindexed_dim_sizes_equal raise ValueError( E ValueError: cannot reindex or align along dimension 'bc' because of conflicting dimension sizes: {10, 6} (note: an index is found along that dimension with size=10)

Anything else we need to know?

No response

Environment

What is your issue?

It's failing during setup: https://github.com/pydata/xarray/runs/6424624397?check_suite_focus=true

``` · Discovering benchmarks ·· Uninstalling from conda-py3.8-bottleneck-dask-distributed-flox-netcdf4-numpy-numpy_groupies-pandas-scipy-sparse ·· Building dd20d07f for conda-py3.8-bottleneck-dask-distributed-flox-netcdf4-numpy-numpy_groupies-pandas-scipy-sparse ·· Error running /home/runner/work/xarray/xarray/asv_bench/.asv/env/e8ce5703538597037a298414451d04d2/bin/python -mpip wheel --no-deps --no-index -w /home/runner/work/xarray/xarray/asv_bench/.asv/env/e8ce5703538597037a298414451d04d2/asv-build-cache/dd20d07f4057a9e29222ca132c36cbaaf3fbb242 /home/runner/work/xarray/xarray/asv_bench/.asv/env/e8ce5703538597037a298414451d04d2/project (exit status 1) STDOUT --------> Processing /home/runner/work/xarray/xarray/asv_bench/.asv/env/e8ce5703538597037a298414451d04d2/project STDERR --------> ERROR: Some build dependencies for file:///home/runner/work/xarray/xarray/asv_bench/.asv/env/e8ce5703538597037a298414451d04d2/project are missing: 'setuptools_scm[toml]>=3.4', 'setuptools_scm_git_archive'.

·· Failed: trying different commit/environment ·· Uninstalling from conda-py3.8-bottleneck-dask-distributed-flox-netcdf4-numpy-numpy_groupies-pandas-scipy-sparse ·· Building c34ef8a6 for conda-py3.8-bottleneck-dask-distributed-flox-netcdf4-numpy-numpy_groupies-pandas-scipy-sparse ·· Error running /home/runner/work/xarray/xarray/asv_bench/.asv/env/e8ce5703538597037a298414451d04d2/bin/python -mpip wheel --no-deps --no-index -w /home/runner/work/xarray/xarray/asv_bench/.asv/env/e8ce5703538597037a298414451d04d2/asv-build-cache/c34ef8a60227720724e90aa11a6266c0026a812a /home/runner/work/xarray/xarray/asv_bench/.asv/env/e8ce5703538597037a298414451d04d2/project (exit status 1) STDOUT --------> Processing /home/runner/work/xarray/xarray/asv_bench/.asv/env/e8ce5703538597037a298414451d04d2/project STDERR --------> ERROR: Some build dependencies for file:///home/runner/work/xarray/xarray/asv_bench/.asv/env/e8ce5703538597037a298414451d04d2/project are missing: 'setuptools_scm[toml]>=3.4', 'setuptools_scm_git_archive'. ```

Is your feature request related to a problem?

polyval does not handle dask inputs well.

```python nt = 8772 // 4 ny = 489 nx = 655

chunks like the data is stored on disk

small in time, big in space

because the chunk sizes are -1 along lat, lon;

reshaping this array to (time, latlon) prior to fitting is pretty cheap

chunks = (8, -1, -1)

da = xr.DataArray( dask.array.random.random((nt, ny, nx), chunks=chunks), dims=("ocean_time", "eta_rho", "xi_rho"), )

dim = "ocean_time" deg = 1

p = da.polyfit(dim="ocean_time", deg=1, skipna=False)

create a chunked version of the "ocean_time" dimension

chunked_dim = xr.DataArray( dask.array.from_array(da[dim].data, chunks=da.chunksizes[dim]), dims=dim, name=dim ) xr.polyval(chunked_dim, p.polyfit_coefficients) ```

Describe the solution you'd like

Here's a partial solution. It does not handle datetime inputs (polyval handles this using get_clean_interp_index which computes dask inputs). But I've replaced the call to np.vander and used xr.dot.

```python def polyval(coord, coeffs, degree_dim="degree"): x = coord.data

deg_coord = coeffs[degree_dim]
N = int(deg_coord.max()) + 1

lhs = xr.DataArray(
    np.stack([x ** (N - 1 - i) for i in range(N)], axis=1),
    dims=(coord.name, degree_dim),
    coords={coord.name: coord, degree_dim: np.arange(deg_coord.max() + 1)[::-1]},
)
return xr.dot(lhs, coeffs, dims=degree_dim)

polyval(chunked_dim, p.polyfit_coefficients) ```

This looks like what I expected

cc @aulemahal

Describe alternatives you've considered

No response

Additional context

No response

What is your issue?

We have some stale issue but not that many.

Can we restrict the bot to only issues that are untagged, or tagged as "usage question" or are not assigned to a "project" instead? This might reduce a lot of the noise.

What happened?

``` python import numpy as np import xarray as xr

ds = xr.Dataset( {"foo": (("x", "y", "z"), np.ones((3, 4, 2)))}, {"x": ["a", "b", "c"], "y": [1, 2, 3, 4]}, ) expected = ds.sum("z") stacked = ds.stack(space=["x", "y"])

broadcasted, _ = xr.broadcast(stacked, stacked.space)

stacked.sum("z").unstack("space") # works broadcasted.sum("z").unstack("space") # error ```

```

ValueError Traceback (most recent call last) Input In [13], in <module> 10 broadcasted, _ = xr.broadcast(stacked, stacked.space) 11 stacked.sum("z").unstack("space") ---> 12 broadcasted.sum("z").unstack("space")

File ~/work/python/xarray/xarray/core/dataset.py:4332, in Dataset.unstack(self, dim, fill_value, sparse) 4330 non_multi_dims = set(dims) - set(stacked_indexes) 4331 if non_multi_dims: -> 4332 raise ValueError( 4333 "cannot unstack dimensions that do not " 4334 f"have exactly one multi-index: {tuple(non_multi_dims)}" 4335 ) 4337 result = self.copy(deep=False) 4339 # we want to avoid allocating an object-dtype ndarray for a MultiIndex, 4340 # so we can't just access self.variables[v].data for every variable. 4341 # We only check the non-index variables. 4342 # https://github.com/pydata/xarray/issues/5902

ValueError: cannot unstack dimensions that do not have exactly one multi-index: ('space',) ```

What did you expect to happen?

This should work.

Minimal Complete Verifiable Example

No response

Relevant log output

No response

Anything else we need to know?

No response

Environment

xarray main after the flexible indexes refactor

What happened?

``` python import xarray as xr

xr.where(xr.DataArray([1, 2, 3]) > 0, 1, 0) ```

fails with

`` 1809 if keep_attrs is True: 1810 # keep the attributes of x, the second parameter, by default to 1811 # be consistent with thewheremethod ofDataArrayandDataset` -> 1812 keep_attrs = lambda attrs, context: attrs[1] 1814 # alignment for three arguments is complicated, so don't support it yet 1815 return apply_ufunc( 1816 duck_array_ops.where, 1817 cond, (...) 1823 keep_attrs=keep_attrs, 1824 )

IndexError: list index out of range ```

The workaround is to pass keep_attrs=False

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

Relevant log output

No response

Anything else we need to know?

No response

Environment

xarray 2022.3.0

version rasterio 1.1.1 py36h900e953_0 conda-forge

``` =================================== FAILURES =================================== ___ TestRasterio.testrasterio_vrt ____

self = <xarray.tests.test_backends.TestRasterio object at 0x7fc8355c8f60>

def test_rasterio_vrt(self):
    import rasterio

    # tmp_file default crs is UTM: CRS({'init': 'epsg:32618'}
    with create_tmp_geotiff() as (tmp_file, expected):
        with rasterio.open(tmp_file) as src:
            with rasterio.vrt.WarpedVRT(src, crs="epsg:4326") as vrt:
                expected_shape = (vrt.width, vrt.height)
                expected_crs = vrt.crs
                expected_res = vrt.res
                # Value of single pixel in center of image
                lon, lat = vrt.xy(vrt.width // 2, vrt.height // 2)

              expected_val = next(vrt.sample([(lon, lat)]))

xarray/tests/test_backends.py:3966:

/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/rasterio/sample.py:43: in sample_gen data = read(indexes, window=window, masked=masked, boundless=True)

??? E ValueError: WarpedVRT does not permit boundless reads

rasterio/_warp.pyx:978: ValueError ```

What happened?

Really hard to describe in words =)

concat = xr.concat([da.isel(time=0), da.isel(time=[1])], dim="time") xr.align(concat, da, dim="time")

fails when concat and da should be identical. This is causing failures in cf-xarray:https://github.com/xarray-contrib/cf-xarray/issues/319

cc @benbovy

What did you expect to happen?

No response

Minimal Complete Verifiable Example

```Python import numpy as np import xarray as xr

time = xr.DataArray( np.array( ["2013-01-01T00:00:00.000000000", "2013-01-01T06:00:00.000000000"], dtype="datetime64[ns]", ), dims="time", name="time", )

da = time concat = xr.concat([da.isel(time=0), da.isel(time=[1])], dim="time") xr.align(da, concat, join="exact") # works

da = xr.DataArray(np.ones(time.shape), dims="time", coords={"time": time}) concat = xr.concat([da.isel(time=0), da.isel(time=[1])], dim="time") xr.align(da, concat, join="exact") ```

Relevant log output

```

ValueError Traceback (most recent call last) Input In [27], in <module> 17 da = xr.DataArray(np.ones(time.shape), dims="time", coords={"time": time}) 18 concat = xr.concat([da.isel(time=0), da.isel(time=[1])], dim="time") ---> 19 xr.align(da, concat, join="exact")

File ~/work/python/xarray/xarray/core/alignment.py:761, in align(join, copy, indexes, exclude, fill_value, *objects) 566 """ 567 Given any number of Dataset and/or DataArray objects, returns new 568 objects with aligned indexes and dimension sizes. (...) 751 752 """ 753 aligner = Aligner( 754 objects, 755 join=join, (...) 759 fill_value=fill_value, 760 ) --> 761 aligner.align() 762 return aligner.results

File ~/work/python/xarray/xarray/core/alignment.py:549, in Aligner.align(self) 547 self.find_matching_unindexed_dims() 548 self.assert_no_index_conflict() --> 549 self.align_indexes() 550 self.assert_unindexed_dim_sizes_equal() 552 if self.join == "override":

File ~/work/python/xarray/xarray/core/alignment.py:395, in Aligner.align_indexes(self) 393 if need_reindex: 394 if self.join == "exact": --> 395 raise ValueError( 396 "cannot align objects with join='exact' where " 397 "index/labels/sizes are not equal along " 398 "these coordinates (dimensions): " 399 + ", ".join(f"{name!r} {dims!r}" for name, dims in key[0]) 400 ) 401 joiner = self._get_index_joiner(index_cls) 402 joined_index = joiner(matching_indexes)

ValueError: cannot align objects with join='exact' where index/labels/sizes are not equal along these coordinates (dimensions): 'time' ('time',) ```

Anything else we need to know?

No response

Environment

xarray main

By switching to numpy_groupies we are vectorizing our groupby reductions. I think we can do the same for groupby's binary ops.

Here's an example array ``` python import numpy as np import xarray as xr

%load_ext memory_profiler

N = 4 * 2000 da = xr.DataArray( np.random.random((N, N)), dims=("x", "y"), coords={"labels": ("x", np.repeat(["a", "b", "c", "d", "e", "f", "g", "h"], repeats=N//8))}, ) ```

Consider this "anomaly" calculation, anomaly defined relative to the group mean

``` python def anom_current(da): grouped = da.groupby("labels") mean = grouped.mean() anom = grouped - mean return anom

```

With this approach, we loop over each group and apply the binary operation: https://github.com/pydata/xarray/blob/a1635d324753588e353e4e747f6058936fa8cf1e/xarray/core/computation.py#L502-L525

This saves some memory, but becomes slow for large number of groups.

We could instead do def anom_vectorized(da): mean = da.groupby("labels").mean() mean_expanded = mean.sel(labels=da.labels) anom = da - mean_expanded return anom

Now we are faster, but construct an extra array as big as the original array (I think this is an OK tradeoff). ``` %timeit anom_current(da)

1.4 s ± 20.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit anom_vectorized(da)

937 ms ± 5.26 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```

(I haven't experimented with dask yet, so the following is just a theory).

I think the real benefit comes with dask. Depending on where the groups are located relative to chunking, we could end up creating a lot of tiny chunks by splitting up existing chunks. With the vectorized approach we can do better.

Ideally we would reindex the "mean" dask array with a numpy-array-of-repeated-ints such that the chunking of mean_expanded exactly matches the chunking of da along the grouped dimension.

~In practice, dask.array.take doesn't allow specifying "output chunks" so we'd end up chunking "mean_expanded" based on dask's automatic heuristics, and then rechunking again for the binary operation.~

Thoughts?

cc @rabernat

What is your issue?

I merged #5950 but forgot that it included some backward-incompatible changes (Sorry! this came up in https://github.com/pydata/xarray/pull/6403 thanks to @mathause for spotting.)

1. Arguments like keep_attrs, axis are now keyword-only.
2. Some reductions had the 3rd position arg as keep_attrs and in other cases it was axis.

These have been standardized now, and only dim is accepted without kwarg-name.

@pydata/xarray Should we add a deprecation cycle (https://github.com/pydata/xarray/issues/5531)? Point (2) above will make it a little messy.

At the very least we should add a deprecation notice before releasing.

What happened?

After vectorized indexing a DataArray with dim xby a DataArray z, we get a DataArray with dim z and x as non-dim coordinate. But x is still an IndexVariable, not a normal variable.

What did you expect to happen?

x should be a normal variable.

Minimal Complete Verifiable Example

```python import xarray as xr xr.set_options(display_style="text")

da = xr.DataArray([1, 2, 3], dims="x", coords={"x": [0, 1, 2]}) idxr = xr.DataArray([1], dims="z", name="x", coords={"z": ("z", ["a"])}) da.sel(x=idxr) ```

<xarray.DataArray (z: 1)> array([2]) Coordinates: x (z) int64 1 * z (z) <U1 'a'

x is a non-dim coordinate but is backed by a IndexVariable with the wrong name! python da.sel(x=idxr).x.variable

<xarray.IndexVariable 'z' (z: 1)> array([1])

Relevant log output

No response

Anything else we need to know?

No response

Environment

xarray main but this bug was present prior to the explicit indexes refactor.

What is your issue?

The zarr backend has a list of "valid" encodings that needs to be updated any time zarr adds something new (e.g. https://github.com/pydata/xarray/pull/6348)

https://github.com/pydata/xarray/blob/53172cb1e03a98759faf77ef48efaa64676ad24a/xarray/backends/zarr.py#L215-L234

Can we get rid of this? I don't know the backends code well, but won't all our encoding parameters have been removed by this stage? The raise_on_invalid kwarg suggests so.

@tomwhite points out that zarr will raise a warning: ``` python

zarr.create((1,), blah=1) /Users/tom/miniconda/envs/sgkit-3.8/lib/python3.8/site-packages/zarr/creation.py:221: UserWarning: ignoring keyword argument 'blah' warn('ignoring keyword argument %r' % k) <zarr.core.Array (1,) float64> ```

What happened?

This test is failing with a 404 error:

https://github.com/pydata/xarray/blob/95bb9ae4233c16639682a532c14b26a3ea2728f3/xarray/tests/test_backends.py#L4778-L4802

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

Relevant log output

No response

Anything else we need to know?

No response

Environment

N/A

Is your feature request related to a problem?

It seems like we should have a test to make sure our dependencies are specified correctly.

Describe the solution you'd like

For instance we could add a step to the release workflow: https://github.com/pydata/xarray/blob/b09de8195a9e22dd35d1b7ed608ea15dad0806ef/.github/workflows/pypi-release.yaml#L34-L43

after twine check where we pip install and then try to import xarray.

Alternatively we could have another test config in our regular CI to build + import.

Thoughts? Is this excessive for a somewhat rare problem?

Describe alternatives you've considered

No response

Additional context

No response

Is your feature request related to a problem? Please describe.

The following doesn't work ``` python import xarray as xr

da = xr.DataArray([[1,2],[1,2]], dims=("x", "y")) da.stack(flat=...) ```

Describe the solution you'd like This could be equivalent to python da.stack(flat=da.dims)

I think using ds.dims it should be fine for datasets too.

Now that our backends work is well underway, it seems like a good time to discuss deprecating the open_rasterio function and outsourcing it to rioxarray (which would then implement support for the new entrypoint).

From @jhamman's comment

There's also the question of where the rasterio backend should live once #1970 is implemented. I think we can make a pretty strong argument that it should move out of xarray and into a 3rd party package. Perhaps rasterio itself, or, more likely something like rio-xarray (cc @snowman2). See #3166 for a demo of how this may work.

As you can see, there are a few big questions up in the air right now. Work on #1970 will begin this summer so now is a great time to game out the various options for the rasterio backend.

I am in favour of doing this. I see no reason why there should be two slightly divergent versions of the same function, and (anecdotally) the rioxarray version seems to be more full-featured and regularly improved.

@pydata/xarray What do you think?

If we agree to do this (and rioxarray agrees), we could put this as a "deliverable" in our NASA proposal. This work would be totally relevant to that call.

Related: #4142

cc @snowman2 @scottyhq @JessicaS11 @WeatherGod

The current RTD build is failing in plotting.rst

`` sphinx.errors.SphinxParallelError: RuntimeError: Non Expected exception in/home/docs/checkouts/readthedocs.org/user_builds/xray/checkouts/latest/doc/user-guide/plotting.rst` line None

Sphinx parallel build error: RuntimeError: Non Expected exception in /home/docs/checkouts/readthedocs.org/user_builds/xray/checkouts/latest/doc/user-guide/plotting.rst line None [IPKernelApp] WARNING | Parent appears to have exited, shutting down. ```

As far as I know, there's no way to see what options are enabled.

It'd be nice to add either xr.get_options or xr.show_options or xr.options so that users can see what is enabled.

I think it would be nice to have a one-line description under each option

"arithmetic_join": "outer" controls DataArray/Dataset alignment in binary operations.

Issues I've run into working with @friedrichknuth at the Pangeo meeting trying to use sparse

1. .plot() doesn't work. We need to convert to dense before plotting
2. .values doesn't work. Raises an error about explicitly calling todense()
3. align doesn't seem to work.

MCVE Code Sample

python da1 = xr.DataArray([1, 2, 3], dims='x', coords={'x': [0, 1, 2]}) da2 = xr.DataArray([3, 4, 5], dims='x', coords={'x': [2, 3, 4]}) xr.combine_by_coords([da1, da2])

```

ValueError Traceback (most recent call last) <ipython-input-133-d27216ba5688> in <module> 1 da1 = xr.DataArray([1, 2, 3], dims='x', coords={'x': [0, 1, 2]}) 2 da2 = xr.DataArray([3, 4, 5], dims='x', coords={'x': [2, 3, 4]}) ----> 3 xr.combine_by_coords([da1, da2])

~/work/python/xarray/xarray/core/combine.py in combine_by_coords(datasets, compat, data_vars, coords, fill_value, join) 619 compat=compat, 620 fill_value=fill_value, --> 621 join=join, 622 ) 623

~/work/python/xarray/xarray/core/merge.py in merge(objects, compat, join, fill_value) 588 ) 589 --> 590 obj = obj.to_dataset() if isinstance(obj, DataArray) else obj 591 dict_like_objects.append(obj) 592

~/work/python/xarray/xarray/core/dataarray.py in to_dataset(self, dim, name) 478 return self._to_dataset_split(dim) 479 else: --> 480 return self._to_dataset_whole(name) 481 482 @property

~/work/python/xarray/xarray/core/dataarray.py in _to_dataset_whole(self, name, shallow_copy) 426 if name is None: 427 raise ValueError( --> 428 "unable to convert unnamed DataArray to a " 429 "Dataset without providing an explicit name" 430 )

ValueError: unable to convert unnamed DataArray to a Dataset without providing an explicit name ```

To get what I want, I need xr.combine_by_coords([da1.to_dataset(name='a'), da2.to_dataset(name='a')]).a

I think the issue is that the code uses to_dataset instead of _to_temp_dataset

Like Rolling.construct this would create a new reshaped array with extra dimensions that would usually be reduced over. It would be a light wrapper over Variable._coarsen_reshape

It make https://github.com/pydata/xarray/discussions/5119, and #2419 really easy without having to go through a MultiIndex+unstack operation which is somewhat arcane. python ds.coarsen(time=12).construct(dim="year").rename({"time": "month"})

Is your feature request related to a problem? Please describe. xarray currently doesn't allow plotting against coordinates with string labels for example.

Describe the solution you'd like Use matplotlib's categorical axis support. Example: https://matplotlib.org/gallery/lines_bars_and_markers/categorical_variables.html

This may be the only place a change is required: https://github.com/pydata/xarray/blob/4e893317240ed1a80e65ea2de107e9179bb65446/xarray/plot/utils.py#L572-L608

This should handle multiple DataArrays and Datasets.

Implemented in #3276 as Dataset.unify_chunks and DataArray.unify_chunks

I think we should issue a patch release soon.

• [x] xarray.map_blocks will break with new dask,distributed releases this week since we don't have a released version with https://github.com/pydata/xarray/pull/5271
• [x] #5296: fix for regression reported in #5295
• [x] #5300: fix the error message in #5291 so that it is more helpful. It's a key entry point for new users.
• [x] #5299 RTD build is broken
• [ ] #5329 should throw FileNotFoundError error on nonexistent file?

Nice to have - [ ] #5252 zarr region - [ ] #5045 assigning to subset - [x] #5239 drop_duplicates - [x] #5288 invalid reference date error

Thoughts? Volunteers?

The RTD build is failing on all PRs with

Sphinx parallel build error: nbsphinx.NotebookError: UndefinedError in examples/ERA5-GRIB-example.ipynb: 'nbformat.notebooknode.NotebookNode object' has no attribute 'tags'

As discussed in the meeting, we should issue a release soon with the new backend refactor and the new docs theme.

Here's a list of blockers: - [x] #5231 - [x] #5073 - [x] #5235

Would be nice and look done: - [x] #5244 - [x] #5258 - [x] #5101 - [x] ~#4866~ (we should let this sit on master for a while to find bugs) - [x] #4902 - [x] ~#4972~ (this should probably also sit on master for a while) - [x] #5227 - [x] #4740 - [x] #5149

Somewhat important, but no PR yet: - [x] ~#5175~ (as pointed out by @shoyer, this is really a new feature, not a regression, it can wait)

@TomNicholas and @alexamici volunteered to handle this. I can be online at release time to help with things if needed.

Release instructions are here: https://github.com/pydata/xarray/blob/master/HOW_TO_RELEASE.md

IIRC they'll need to be added to the PyPI list and RTD list.

Any thoughts on adding

1. .chunksize or .nbytes_chunk
2. .ntasks : this would be len(obj.__dask_graph__())

This came up in #2844 by @DWesl (see also #3689)

Currently we have decode_coords which sets variables named in attrs["coordinates"] as coordinate variables.

There are a number of other CF attributes that can contain variable names. 1. bounds 1. grid_mapping 1. ancillary_variables 1. cell_measures 1. maybe more?

As in #3689 it's hard to see why a lot of these variables named in these attributes would be useful as "data variables".

Question: Should we allow decode_coords to control whether variables mentioned in these attributes are set as coordinate variables?

cc @jthielen

What happened: The formatting of section headings "for users", "community" etc. has changed: https://xarray.pydata.org/en/latest/

What happened:

open_dataset doesn't replace ~ any more

What you expected to happen:

should replace ~ for backward compatibility

Minimal Complete Verifiable Example:

```python import xarray as xr

da = xr.DataArray([1, 2, 3]) da.to_netcdf("~/bug_report.nc")

xr.open_dataarray("~/bug_report.nc") ```

Anything else we need to know?: works on 0.17.0, fails on master

Full log here: https://dev.azure.com/xarray/xarray/_build/results?buildId=3023&view=logs&jobId=2280efed-fda1-53bd-9213-1fa8ec9b4fa8&j=2280efed-fda1-53bd-9213-1fa8ec9b4fa8&t=175181ee-1928-5a6b-f537-168f7a8b7c2d

Here are three of the errors:

/usr/share/miniconda/envs/xarray-tests/lib/python3.8/site-packages/sparse/_coo/umath.py:739: SystemError _ test_variable_method[obj.where(*(), **{'cond': <xarray.Variable (x: 10, y: 5)>\n<COO: shape=(10, 5), dtype=bool, nnz=3, fill_value=False>})-True] _ TypeError: expected dtype object, got 'numpy.dtype[uint64]'

`` def _match_coo(*args, **kwargs): """ Matches the coordinates for any number of input :obj:COO` arrays. Equivalent to "sparse" broadcasting for all arrays.

    Parameters
    ----------
    args : Tuple[COO]
        The input :obj:`COO` arrays.
    return_midx : bool
        Whether to return matched indices or matched arrays. Matching
        only supported for two arrays. ``False`` by default.
    cache : dict
        Cache of things already matched. No cache by default.

    Returns
    -------
    matched_idx : List[ndarray]
        The indices of matched elements in the original arrays. Only returned if
        ``return_midx`` is ``True``.
    matched_arrays : List[COO]
        The expanded, matched :obj:`COO` objects. Only returned if
        ``return_midx`` is ``False``.
    """
    from .core import COO
    from .common import linear_loc

    cache = kwargs.pop("cache", None)
    return_midx = kwargs.pop("return_midx", False)
    broadcast_shape = kwargs.pop("broadcast_shape", None)

    if kwargs:
        linear = [idx[s] for idx, s in zip(linear, sorted_idx)]

      matched_idx = _match_arrays(*linear)

E SystemError: CPUDispatcher(<function _match_arrays at 0x7f66b6272af0>) returned a result with an error set

```

``` ____ testdask_token ______

@requires_dask
def test_dask_token():
    import dask

    s = sparse.COO.from_numpy(np.array([0, 0, 1, 2]))

    # https://github.com/pydata/sparse/issues/300
    s.__dask_tokenize__ = lambda: dask.base.normalize_token(s.__dict__)

    a = DataArray(s)
    t1 = dask.base.tokenize(a)
    t2 = dask.base.tokenize(a)
    t3 = dask.base.tokenize(a + 1)
    assert t1 == t2
    assert t3 != t2
    assert isinstance(a.data, sparse.COO)

    ac = a.chunk(2)
    t4 = dask.base.tokenize(ac)
    t5 = dask.base.tokenize(ac + 1)
    assert t4 != t5

  assert isinstance(ac.data._meta, sparse.COO)

E AssertionError: assert False E + where False = isinstance(array([], dtype=int64), <class 'sparse._coo.core.COO'>) E + where array([], dtype=int64) = dask.array<xarray-\<this-array>, shape=(4,), dtype=int64, chunksize=(2,), chunktype=numpy.ndarray>._meta E + where dask.array<xarray-\<this-array>, shape=(4,), dtype=int64, chunksize=(2,), chunktype=numpy.ndarray> = <xarray.DataArray (dim_0: 4)>\ndask.array<xarray-\<this-array>, shape=(4,), dtype=int64, chunksize=(2,), chunktype=numpy.ndarray>\nDimensions without coordinates: dim_0.data E + and <class 'sparse._coo.core.COO'> = sparse.COO ```

Just as a reminder.

Right now all rolling operations with dask arrays use .construct().reduce().

Quoting @spencerkclark email:

Somewhat related to this issue, would others on the core team be open to making a new xarray release soon? In addition to the license change, this cftime update unintentionally introduced a change* that broke our encoding logic for cftime dates in a subtle way. I recently fixed this in pydata/xarray#4871, but until a new xarray release is made, this issue makes it dangerous to use the latest non-GPL-licensed version of cftime.

These three seem like high-priority fixes. Thoughts?

• [x] #4884
• [x] #4893
• [ ] #4879 ?
• [x] #4911
• [x] #4179 / #4907

Nice to have

• [ ] #4740

I think it would be nice to add a quiver plot function. I got this far in my current project:

``` python @xarray.plot.dataset_plot._dsplot def quiver(ds, x, y, ax, u, v, **kwargs): from xarray import broadcast

if x is None or y is None or u is None or v is None:
    raise ValueError("Must specify x, y, u, v for quiver plots.")

# matplotlib autoscaling algorithm
scale = kwargs.pop("scale", None)
if scale is None:
    npts = ds.dims[x] * ds.dims[y]
    # crude auto-scaling
    # scale is typical arrow length as a multiple of the arrow width
    scale = (
        1.8 * ds.to_array().median().values * np.maximum(10, np.sqrt(npts))
    )  # / span

ds = ds.squeeze()
x, y, u, v = broadcast(ds[x], ds[y], ds[u], ds[v])

# no support for these kwargs right now.
kwargs.pop("cmap_params")
kwargs.pop("hue")
kwargs.pop("hue_style")
hdl = ax.quiver(x.values, y.values, u.values, v.values, scale=scale, **kwargs)

return hdl

```

The autoscaling logic is quite crude; I tried to copy what matplotlib does but got somewhat confused. To get faceting to work properly, we'll need to estimate scale around here: https://github.com/pydata/xarray/blob/a36d0a1d4657c848dcdd76d0ecb9c783ad464057/xarray/plot/facetgrid.py#L339-L343

Downstream issue: https://github.com/xarray-contrib/cf-xarray/issues/57 Example: no reprs displayed in https://cf-xarray.readthedocs.io/en/latest/examples/introduction.html

@benbovy's diagnosis:

It looks like bootstrap 4 (used by sphinx-book-theme) forces all html elements with hidden attributes to be actually hidden (source), so the hack in pydata/xarray#4053 does not work here (the result is even worse). I guess that a workaround would be to add some custom CSS such as .xr-wrap { display: block !important }, assuming that custom CSS is loaded after Bootstrap's CSS. Not ideal, though, it looks like a hack on top of another hack.

RTD now supports using mamba instead of conda using a feature flag : https://docs.readthedocs.io/en/latest/guides/feature-flags.html#available-flags

Maybe it's worth seeing if this will reduce docs build time on CI?

Looks like something went wrong when #4408 was merged (https://github.com/pydata/xarray/commit/23dc2fc9f2785c348ff821bf2da61dfa2206d283)

pytest -n 4 test_sparse.py results in

``` ====================================================================================== ERRORS ====================================================================================== ___________ ERROR collecting gw0 ___________ Different tests were collected between gw1 and gw0. The difference is: --- gw1

+++ gw0

@@ -11,7 +11,7 @@

xarray/tests/test_sparse.py::test_variable_method[obj.any((), {})-False] xarray/tests/test_sparse.py::test_variable_method[obj.astype((), {'dtype': <class 'int'>})-True] xarray/tests/test_sparse.py::test_variable_method[obj.clip(*(), {'min': 0, 'max': 1})-True] -xarray/tests/test_sparse.py::test_variable_method[obj.coarsen((), {'windows': {'x': 2}, 'func': <function sum at 0x7fc7303f7d08>})-True] +xarray/tests/test_sparse.py::test_variable_method[obj.coarsen((), {'windows': {'x': 2}, 'func': <function sum at 0x7f6009fa4d08>})-True] xarray/tests/test_sparse.py::test_variable_method[obj.compute(*(), {})-True] xarray/tests/test_sparse.py::test_variable_method[obj.conj((), {})-True] xarray/tests/test_sparse.py::test_variable_method[obj.copy((), **{})-True] @@ -49,7 +49,7 @@

xarray/tests/test_sparse.py::test_variable_method[obj.prod((), {})-False] xarray/tests/test_sparse.py::test_variable_method[obj.quantile((), {'q': 0.5})-True] xarray/tests/test_sparse.py::test_variable_method[obj.rank(*(), {'dim': 'x'})-False] -xarray/tests/test_sparse.py::test_variable_method[obj.reduce((), {'func': <function sum at 0x7fc7303f7d08>, 'dim': 'x'})-True] +xarray/tests/test_sparse.py::test_variable_method[obj.reduce((), {'func': <function sum at 0x7f6009fa4d08>, 'dim': 'x'})-True] xarray/tests/test_sparse.py::test_variable_method[obj.rolling_window(*(), {'dim': 'x', 'window': 2, 'window_dim': 'x_win'})-True] xarray/tests/test_sparse.py::test_variable_method[obj.shift((), {'x': 2})-True] xarray/tests/test_sparse.py::test_variable_method[obj.std((), **{})-False] @@ -144,11 +144,11 @@

xarray/tests/test_sparse.py::test_dataarray_method[obj.median((), {})-False] xarray/tests/test_sparse.py::test_dataarray_method[obj.min((), {})-False] xarray/tests/test_sparse.py::test_dataarray_method[obj.notnull(*(), {})-False] -xarray/tests/test_sparse.py::test_dataarray_method[obj.pipe((<function sum at 0x7fc7303f7d08>,), {'axis': 1})-True] +xarray/tests/test_sparse.py::test_dataarray_method[obj.pipe((<function sum at 0x7f6009fa4d08>,), {'axis': 1})-True] xarray/tests/test_sparse.py::test_dataarray_method[obj.prod(*(), {})-False] xarray/tests/test_sparse.py::test_dataarray_method[obj.quantile((), {'q': 0.5})-False] xarray/tests/test_sparse.py::test_dataarray_method[obj.rank(('x',), {})-False] -xarray/tests/test_sparse.py::test_dataarray_method[obj.reduce(*(<function sum at 0x7fc7303f7d08>,), {'dim': 'x'})-False] +xarray/tests/test_sparse.py::test_dataarray_method[obj.reduce((<function sum at 0x7f6009fa4d08>,), {'dim': 'x'})-False] xarray/tests/test_sparse.py::test_dataarray_method[obj.reindex_like((<xarray.DataArray 'test' (x: 10, y: 5)>\n<COO: shape=(10, 5), dtype=float64, nnz=5, fill_value=0.0>\nCoordinates:\n * x (x) float64 0.5 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5\n * y (y) float64 0.5 1.5 2.5 3.5 4.5,), {})-True] xarray/tests/test_sparse.py::test_dataarray_method[obj.roll(*(), {'x': 2})-True] xarray/tests/test_sparse.py::test_dataarray_method[obj.sel((), *{'x': [0, 1, 2], 'y': [2, 3]})-True] ___________ ERROR collecting gw2 ___________ Different tests were collected between gw1 and gw2. The difference is: --- gw1

+++ gw2

@@ -11,7 +11,7 @@

xarray/tests/test_sparse.py::test_variable_method[obj.any((), {})-False] xarray/tests/test_sparse.py::test_variable_method[obj.astype((), {'dtype': <class 'int'>})-True] xarray/tests/test_sparse.py::test_variable_method[obj.clip(*(), {'min': 0, 'max': 1})-True] -xarray/tests/test_sparse.py::test_variable_method[obj.coarsen((), {'windows': {'x': 2}, 'func': <function sum at 0x7fc7303f7d08>})-True] +xarray/tests/test_sparse.py::test_variable_method[obj.coarsen((), {'windows': {'x': 2}, 'func': <function sum at 0x7f657c314d08>})-True] xarray/tests/test_sparse.py::test_variable_method[obj.compute(*(), {})-True] xarray/tests/test_sparse.py::test_variable_method[obj.conj((), {})-True] xarray/tests/test_sparse.py::test_variable_method[obj.copy((), **{})-True] @@ -49,7 +49,7 @@

xarray/tests/test_sparse.py::test_variable_method[obj.prod((), {})-False] xarray/tests/test_sparse.py::test_variable_method[obj.quantile((), {'q': 0.5})-True] xarray/tests/test_sparse.py::test_variable_method[obj.rank(*(), {'dim': 'x'})-False] -xarray/tests/test_sparse.py::test_variable_method[obj.reduce((), {'func': <function sum at 0x7fc7303f7d08>, 'dim': 'x'})-True] +xarray/tests/test_sparse.py::test_variable_method[obj.reduce((), {'func': <function sum at 0x7f657c314d08>, 'dim': 'x'})-True] xarray/tests/test_sparse.py::test_variable_method[obj.rolling_window(*(), {'dim': 'x', 'window': 2, 'window_dim': 'x_win'})-True] xarray/tests/test_sparse.py::test_variable_method[obj.shift((), {'x': 2})-True] xarray/tests/test_sparse.py::test_variable_method[obj.std((), **{})-False] @@ -118,7 +118,7 @@

xarray/tests/test_sparse.py::test_dataarray_method[obj.sel((), {'x': [0, 1, 2]})-True] xarray/tests/test_sparse.py::test_dataarray_method[obj.shift((), {})-True] xarray/tests/test_sparse.py::test_dataarray_method[obj.sortby(*('x',), {'ascending': False})-True] -xarray/tests/test_sparse.py::test_dataarray_method[obj.stack((), {'z': {'y', 'x'}})-True] +xarray/tests/test_sparse.py::test_dataarray_method[obj.stack((), {'z': {'x', 'y'}})-True] xarray/tests/test_sparse.py::test_dataarray_method[obj.transpose(*(), {})-True] xarray/tests/test_sparse.py::test_dataarray_method[obj.broadcast_equals((<xarray.Variable (x: 10, y: 5)>\n<COO: shape=(10, 5), dtype=float64, nnz=5, fill_value=0.0>,), {})-False] xarray/tests/test_sparse.py::test_dataarray_method[obj.equals((<xarray.Variable (x: 10, y: 5)>\n<COO: shape=(10, 5), dtype=float64, nnz=5, fill_value=0.0>,), **{})-False] @@ -144,11 +144,11 @@

xarray/tests/test_sparse.py::test_dataarray_method[obj.median((), {})-False] xarray/tests/test_sparse.py::test_dataarray_method[obj.min((), {})-False] xarray/tests/test_sparse.py::test_dataarray_method[obj.notnull(*(), {})-False] -xarray/tests/test_sparse.py::test_dataarray_method[obj.pipe((<function sum at 0x7fc7303f7d08>,), {'axis': 1})-True] +xarray/tests/test_sparse.py::test_dataarray_method[obj.pipe((<function sum at 0x7f657c314d08>,), {'axis': 1})-True] xarray/tests/test_sparse.py::test_dataarray_method[obj.prod(*(), {})-False] xarray/tests/test_sparse.py::test_dataarray_method[obj.quantile((), {'q': 0.5})-False] xarray/tests/test_sparse.py::test_dataarray_method[obj.rank(('x',), {})-False] -xarray/tests/test_sparse.py::test_dataarray_method[obj.reduce(*(<function sum at 0x7fc7303f7d08>,), {'dim': 'x'})-False] +xarray/tests/test_sparse.py::test_dataarray_method[obj.reduce((<function sum at 0x7f657c314d08>,), {'dim': 'x'})-False] xarray/tests/test_sparse.py::test_dataarray_method[obj.reindex_like((<xarray.DataArray 'test' (x: 10, y: 5)>\n<COO: shape=(10, 5), dtype=float64, nnz=5, fill_value=0.0>\nCoordinates:\n * x (x) float64 0.5 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5\n * y (y) float64 0.5 1.5 2.5 3.5 4.5,), {})-True] xarray/tests/test_sparse.py::test_dataarray_method[obj.roll(*(), {'x': 2})-True] xarray/tests/test_sparse.py::test_dataarray_method[obj.sel((), *{'x': [0, 1, 2], 'y': [2, 3]})-True] ___________ ERROR collecting gw3 ___________ Different tests were collected between gw1 and gw3. The difference is: --- gw1

+++ gw3

@@ -11,7 +11,7 @@

xarray/tests/test_sparse.py::test_variable_method[obj.any((), {})-False] xarray/tests/test_sparse.py::test_variable_method[obj.astype((), {'dtype': <class 'int'>})-True] xarray/tests/test_sparse.py::test_variable_method[obj.clip(*(), {'min': 0, 'max': 1})-True] -xarray/tests/test_sparse.py::test_variable_method[obj.coarsen((), {'windows': {'x': 2}, 'func': <function sum at 0x7fc7303f7d08>})-True] +xarray/tests/test_sparse.py::test_variable_method[obj.coarsen((), {'windows': {'x': 2}, 'func': <function sum at 0x7f6f284e3d08>})-True] xarray/tests/test_sparse.py::test_variable_method[obj.compute(*(), {})-True] xarray/tests/test_sparse.py::test_variable_method[obj.conj((), {})-True] xarray/tests/test_sparse.py::test_variable_method[obj.copy((), **{})-True] @@ -49,7 +49,7 @@

xarray/tests/test_sparse.py::test_variable_method[obj.prod((), {})-False] xarray/tests/test_sparse.py::test_variable_method[obj.quantile((), {'q': 0.5})-True] xarray/tests/test_sparse.py::test_variable_method[obj.rank(*(), {'dim': 'x'})-False] -xarray/tests/test_sparse.py::test_variable_method[obj.reduce((), {'func': <function sum at 0x7fc7303f7d08>, 'dim': 'x'})-True] +xarray/tests/test_sparse.py::test_variable_method[obj.reduce((), {'func': <function sum at 0x7f6f284e3d08>, 'dim': 'x'})-True] xarray/tests/test_sparse.py::test_variable_method[obj.rolling_window(*(), {'dim': 'x', 'window': 2, 'window_dim': 'x_win'})-True] xarray/tests/test_sparse.py::test_variable_method[obj.shift((), {'x': 2})-True] xarray/tests/test_sparse.py::test_variable_method[obj.std((), **{})-False] @@ -144,11 +144,11 @@

xarray/tests/test_sparse.py::test_dataarray_method[obj.median((), {})-False] xarray/tests/test_sparse.py::test_dataarray_method[obj.min((), {})-False] xarray/tests/test_sparse.py::test_dataarray_method[obj.notnull(*(), {})-False] -xarray/tests/test_sparse.py::test_dataarray_method[obj.pipe((<function sum at 0x7fc7303f7d08>,), {'axis': 1})-True] +xarray/tests/test_sparse.py::test_dataarray_method[obj.pipe((<function sum at 0x7f6f284e3d08>,), {'axis': 1})-True] xarray/tests/test_sparse.py::test_dataarray_method[obj.prod(*(), {})-False] xarray/tests/test_sparse.py::test_dataarray_method[obj.quantile((), {'q': 0.5})-False] xarray/tests/test_sparse.py::test_dataarray_method[obj.rank(('x',), {})-False] -xarray/tests/test_sparse.py::test_dataarray_method[obj.reduce(*(<function sum at 0x7fc7303f7d08>,), {'dim': 'x'})-False] +xarray/tests/test_sparse.py::test_dataarray_method[obj.reduce((<function sum at 0x7f6f284e3d08>,), {'dim': 'x'})-False] xarray/tests/test_sparse.py::test_dataarray_method[obj.reindex_like((<xarray.DataArray 'test' (x: 10, y: 5)>\n<COO: shape=(10, 5), dtype=float64, nnz=5, fill_value=0.0>\nCoordinates:\n * x (x) float64 0.5 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5\n * y (y) float64 0.5 1.5 2.5 3.5 4.5,), {})-True] xarray/tests/test_sparse.py::test_dataarray_method[obj.roll(*(), {'x': 2})-True] xarray/tests/test_sparse.py::test_dataarray_method[obj.sel((), *{'x': [0, 1, 2], 'y': [2, 3]})-True] ================================================================================= warnings summary ================================================================================= /home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/client.py:2 /home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/client.py:2 /home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/client.py:2 /home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/client.py:2 /home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/client.py:2: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working from collections import defaultdict, Iterator

/home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/publish.py:1 /home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/publish.py:1 /home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/publish.py:1 /home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/publish.py:1 /home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/publish.py:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working from collections import MutableMapping

/home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/scheduler.py:2 /home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/scheduler.py:2 /home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/scheduler.py:2 /home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/scheduler.py:2 /home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/scheduler.py:2 /home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/scheduler.py:2 /home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/scheduler.py:2 /home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/scheduler.py:2 /home/deepak/miniconda3/envs/dcpy/lib/python3.7/site-packages/distributed/scheduler.py:2: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working from collections import defaultdict, deque, OrderedDict, Mapping, Set

-- Docs: https://docs.pytest.org/en/latest/warnings.html

```

In IPython, python ds = xr.tutorial.load_dataset("air_temperature") ds.air.groupby("lat").mean? gives output Object `mean` not found.

BUT this does work. python print(ds.air.groupby("lat").mean.__doc__)

`` Reduce this DataArrayGroupBy's data by applyingmean` along some dimension(s).

Parameters

dim : str or sequence of str, optional Dimension(s) over which to apply mean.

... ```

Is your feature request related to a problem? Please describe. I have datasets with lots of non-dim coord variables. Its annoying to search through and look at the dimension coordinates to get an idea of what subset of data I am looking at.

Describe the solution you'd like I think we should show dimension coordinate variables at the top of the coordinates repr.

Example code

python ds = xr.Dataset() ds.coords["as"] = 10 ds["var"] = xr.DataArray(np.ones((10,)), dims="x", coords={"x": np.arange(10)}) ds

Related #4409

A relatively new user here gave me an incomplete list of things they found to be confusing in the docs:

• ~What is xarray's data model?~ opened #2913
• [x] What is split-apply-combine?

• [x] What are dummy dimensions and np.newaxis

• [x] quick-overiew: Index so that you extract a subset!

• [ ] too many references to pandas!

• [x] do I need to know pandas to use xarray?
• [ ] what is the difference between attrs and encoding? How do I use them properly/correctly?
• [ ] why isnull and dropna: why dropna instead of dropnull
• [x] compute docstring doesn't mention computation!
• [x] compute vs persist vs load: what's the difference
• [ ] do I get parallelization for free?
• [ ] do I start a dask cluster first?
• [x] dask page links to shoyers blog but that doesn't explicitly start a cluster.

I'm crossing out things that I've started to fix.

There are a large number of tests in test_dataarray.py and test_dataset.py that use the assert actual.equals(expected) pattern.

We should switch these to be assert_equal(actual, expected) so that we run the invariant checks:

https://github.com/pydata/xarray/blob/acf7d4157ca44f05c85a92d1b914b68738988773/xarray/tests/init.py#L150-L154

Seems like an easy use of regexes for someone that knows them (i.e. not me ;))

I am trying to call dask.optimize on a xarray object before the graph gets too big. But get weird errors. Simple examples below. All examples work if I remove the dask.optimize step.

cc @mrocklin @shoyer

This works with dask arrays:

python a = dask.array.ones((10,5), chunks=(1,3)) a = dask.optimize(a)[0] a.compute()

It works when a dataArray is constructed using a dask array

python da = xr.DataArray(a) da = dask.optimize(da)[0] da.compute()

but fails when creating a DataArray with a numpy array and then chunking it

:man_shrugging: python da = xr.DataArray(a.compute()).chunk({"dim_0": 5}) da = dask.optimize(da)[0] da.compute()

fails with error

``` python

TypeError Traceback (most recent call last) <ipython-input-50-1f16efa19800> in <module> 1 da = xr.DataArray(a.compute()).chunk({"dim_0": 5}) 2 da = dask.optimize(da)[0] ----> 3 da.compute()

~/python/xarray/xarray/core/dataarray.py in compute(self, kwargs) 838 """ 839 new = self.copy(deep=False) --> 840 return new.load(kwargs) 841 842 def persist(self, **kwargs) -> "DataArray":

~/python/xarray/xarray/core/dataarray.py in load(self, kwargs) 812 dask.array.compute 813 """ --> 814 ds = self._to_temp_dataset().load(kwargs) 815 new = self._from_temp_dataset(ds) 816 self._variable = new._variable

~/python/xarray/xarray/core/dataset.py in load(self, kwargs) 659 660 # evaluate all the dask arrays simultaneously --> 661 evaluated_data = da.compute(*lazy_data.values(), kwargs) 662 663 for k, data in zip(lazy_data, evaluated_data):

~/miniconda3/envs/dcpy_updated/lib/python3.7/site-packages/dask/base.py in compute(args, kwargs) 434 keys = [x.dask_keys() for x in collections] 435 postcomputes = [x.dask_postcompute() for x in collections] --> 436 results = schedule(dsk, keys, kwargs) 437 return repack([f(r, a) for r, (f, a) in zip(results, postcomputes)]) 438

~/miniconda3/envs/dcpy_updated/lib/python3.7/site-packages/dask/threaded.py in get(dsk, result, cache, num_workers, pool, kwargs) 79 get_id=_thread_get_id, 80 pack_exception=pack_exception, ---> 81 kwargs 82 ) 83

~/miniconda3/envs/dcpy_updated/lib/python3.7/site-packages/dask/local.py in get_async(apply_async, num_workers, dsk, result, cache, get_id, rerun_exceptions_locally, pack_exception, raise_exception, callbacks, dumps, loads, **kwargs) 484 _execute_task(task, data) # Re-execute locally 485 else: --> 486 raise_exception(exc, tb) 487 res, worker_id = loads(res_info) 488 state["cache"][key] = res

~/miniconda3/envs/dcpy_updated/lib/python3.7/site-packages/dask/local.py in reraise(exc, tb) 314 if exc.traceback is not tb: 315 raise exc.with_traceback(tb) --> 316 raise exc 317 318

~/miniconda3/envs/dcpy_updated/lib/python3.7/site-packages/dask/local.py in execute_task(key, task_info, dumps, loads, get_id, pack_exception) 220 try: 221 task, data = loads(task_info) --> 222 result = _execute_task(task, data) 223 id = get_id() 224 result = dumps((result, id))

~/miniconda3/envs/dcpy_updated/lib/python3.7/site-packages/dask/core.py in _execute_task(arg, cache, dsk) 117 func, args = arg[0], arg[1:] 118 args2 = [_execute_task(a, cache) for a in args] --> 119 return func(*args2) 120 elif not ishashable(arg): 121 return arg

TypeError: string indices must be integers ```

And a different error when rechunking a dask-backed DataArray

python da = xr.DataArray(a).chunk({"dim_0": 5}) da = dask.optimize(da)[0] da.compute()

``` python

IndexError Traceback (most recent call last) <ipython-input-55-d978bbb9e38d> in <module> 1 da = xr.DataArray(a).chunk({"dim_0": 5}) 2 da = dask.optimize(da)[0] ----> 3 da.compute()

~/python/xarray/xarray/core/dataarray.py in compute(self, kwargs) 838 """ 839 new = self.copy(deep=False) --> 840 return new.load(kwargs) 841 842 def persist(self, **kwargs) -> "DataArray":

~/python/xarray/xarray/core/dataarray.py in load(self, kwargs) 812 dask.array.compute 813 """ --> 814 ds = self._to_temp_dataset().load(kwargs) 815 new = self._from_temp_dataset(ds) 816 self._variable = new._variable

~/python/xarray/xarray/core/dataset.py in load(self, kwargs) 659 660 # evaluate all the dask arrays simultaneously --> 661 evaluated_data = da.compute(*lazy_data.values(), kwargs) 662 663 for k, data in zip(lazy_data, evaluated_data):

~/miniconda3/envs/dcpy_updated/lib/python3.7/site-packages/dask/base.py in compute(args, kwargs) 434 keys = [x.dask_keys() for x in collections] 435 postcomputes = [x.dask_postcompute() for x in collections] --> 436 results = schedule(dsk, keys, kwargs) 437 return repack([f(r, a) for r, (f, a) in zip(results, postcomputes)]) 438

~/miniconda3/envs/dcpy_updated/lib/python3.7/site-packages/dask/threaded.py in get(dsk, result, cache, num_workers, pool, kwargs) 79 get_id=_thread_get_id, 80 pack_exception=pack_exception, ---> 81 kwargs 82 ) 83

~/miniconda3/envs/dcpy_updated/lib/python3.7/site-packages/dask/local.py in get_async(apply_async, num_workers, dsk, result, cache, get_id, rerun_exceptions_locally, pack_exception, raise_exception, callbacks, dumps, loads, **kwargs) 484 _execute_task(task, data) # Re-execute locally 485 else: --> 486 raise_exception(exc, tb) 487 res, worker_id = loads(res_info) 488 state["cache"][key] = res

~/miniconda3/envs/dcpy_updated/lib/python3.7/site-packages/dask/local.py in reraise(exc, tb) 314 if exc.traceback is not tb: 315 raise exc.with_traceback(tb) --> 316 raise exc 317 318

~/miniconda3/envs/dcpy_updated/lib/python3.7/site-packages/dask/local.py in execute_task(key, task_info, dumps, loads, get_id, pack_exception) 220 try: 221 task, data = loads(task_info) --> 222 result = _execute_task(task, data) 223 id = get_id() 224 result = dumps((result, id))

~/miniconda3/envs/dcpy_updated/lib/python3.7/site-packages/dask/core.py in _execute_task(arg, cache, dsk) 117 func, args = arg[0], arg[1:] 118 args2 = [_execute_task(a, cache) for a in args] --> 119 return func(*args2) 120 elif not ishashable(arg): 121 return arg

~/miniconda3/envs/dcpy_updated/lib/python3.7/site-packages/dask/array/core.py in concatenate3(arrays) 4305 if not ndim: 4306 return arrays -> 4307 chunks = chunks_from_arrays(arrays) 4308 shape = tuple(map(sum, chunks)) 4309

~/miniconda3/envs/dcpy_updated/lib/python3.7/site-packages/dask/array/core.py in chunks_from_arrays(arrays) 4085 4086 while isinstance(arrays, (list, tuple)): -> 4087 result.append(tuple([shape(deepfirst(a))[dim] for a in arrays])) 4088 arrays = arrays[0] 4089 dim += 1

~/miniconda3/envs/dcpy_updated/lib/python3.7/site-packages/dask/array/core.py in <listcomp>(.0) 4085 4086 while isinstance(arrays, (list, tuple)): -> 4087 result.append(tuple([shape(deepfirst(a))[dim] for a in arrays])) 4088 arrays = arrays[0] 4089 dim += 1

IndexError: tuple index out of range ```

MCVE Code Sample

```python

Your code here

import xarray as xr

ds = xr.tutorial.load_dataset("air_temperature") ds2 = xr.tutorial.load_dataset("air_temperature")

xr.testing.assert_identical(ds, ds2) # passes ds.coarsen(lat=5).mean() xr.testing.assert_identical(ds, ds2) # fails ```

Bug: coarsen reductions are modifying the original dataset by deleting all attributes

``` AssertionErrorTraceback (most recent call last) <ipython-input-21-b0a179f01c99> in <module> 48 xr.testing.assert_identical(ds, ds2) 49 ds.coarsen(lat=5).mean() ---> 50 xr.testing.assert_identical(ds, ds2)

~/work/python/xarray/xarray/testing.py in assert_identical(a, b) 87 assert a.identical(b), formatting.diff_array_repr(a, b, "identical") 88 elif isinstance(a, (Dataset, Variable)): ---> 89 assert a.identical(b), formatting.diff_dataset_repr(a, b, "identical") 90 else: 91 raise TypeError("{} not supported by assertion comparison".format(type(a)))

AssertionError: Left and right Dataset objects are not identical

Differing coordinates: L * lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0 R * lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0 standard_name: latitude long_name: Latitude units: degrees_north axis: Y Differing data variables: L air (time, lat, lon) float32 241.2 242.5 243.5 ... 296.49 296.19 295.69 R air (time, lat, lon) float32 241.2 242.5 243.5 ... 296.49 296.19 295.69 long_name: 4xDaily Air temperature at sigma level 995 units: degK precision: 2 GRIB_id: 11 GRIB_name: TMP var_desc: Air temperature dataset: NMC Reanalysis level_desc: Surface statistic: Individual Obs parent_stat: Other actual_range: [185.16 322.1 ] ```

What happened:

blockwise calls unify_chunks by default but apply_gufunc does not; so we have a regression in apply_ufunc now that we've switched from blockwise to apply_gufunc.

Minimal Complete Verifiable Example:

```python import operator

a = xr.DataArray(np.arange(10), dims=("a")).chunk({"a": 2}) b = xr.DataArray(np.arange(10), dims=("a")).chunk({"a": 4})

xr.apply_ufunc(operator.add, a, b, dask="parallelized", output_dtypes=[a.dtype]).compute() ```

raises ValueError: Dimension `'__loopdim0__'` with different chunksize present

on master but works with 0.16.0

I think we need to do dask_gufunc_kwargs.setdefault("allow_rechunk", True)

If we want to avoid that, we'll need to go through a deprecation cycle.

seen in https://dev.azure.com/xarray/xarray/_build/results?buildId=3365&view=logs&jobId=2280efed-fda1-53bd-9213-1fa8ec9b4fa8&j=2280efed-fda1-53bd-9213-1fa8ec9b4fa8&t=175181ee-1928-5a6b-f537-168f7a8b7c2d

=========================== short test summary info ============================ FAILED xarray/tests/test_plot.py::TestCFDatetimePlot::test_cfdatetime_line_plot FAILED xarray/tests/test_plot.py::TestCFDatetimePlot::test_cfdatetime_pcolormesh_plot FAILED xarray/tests/test_plot.py::TestCFDatetimePlot::test_cfdatetime_contour_plot

e.g. ``` =================================== FAILURES =================================== __ TestCFDatetimePlot.testcfdatetime_line_plot ___

self = <xarray.tests.test_plot.TestCFDatetimePlot object at 0x7f71d66219d0>

def test_cfdatetime_line_plot(self):

E ValueError: setting an array element with a sequence. The requested array would exceed the maximum number of dimension of 1.

/usr/share/miniconda/envs/xarray-tests/lib/python3.8/site-packages/matplotlib/transforms.py:943: ValueError ```

See https://readthedocs.org/projects/xray/builds/

There's no useful information in the logs AFAICT: e.g. https://readthedocs.org/projects/xray/builds/11504571/

It'd be nice to issue a release soon. We should decide if this is a minor 0.15.2 or major 0.16.0

Please edit this list as you wish.

Must-have

• [x] #3936 multiple dims argmin, argmax

• [x] #4019, #4088; MultiIndex to sparse DataArray bug

• [x] #3922 fix dask array handling in idxmax, idxmin
• [x] #4009, #4173 combine_attrs with open_mfdataset
• [x] #3824 transpose coords (if major release)
• [x] #3926 remove old autocombine
• [x] #4135 nD dask arrays in idxmax/min

Nice to have

• [x] #4049, #4094 unstacking merge bug
• [ ] #4022 apply_ufunc meta vectorize
• [ ] #3924 interpolating to coordinates with nans
• [ ] #3925 sel along unindexed 1d coordinates
• [ ] #3993 replace dim with coord in integrate
• [ ] #3594 unit support with pint
• [ ] #4003 mfdataset with zarr
• [x] #3905 length of dataarray reprs
• [x] #3847 error summary for assert_allclose
• [x] #4033 infer_freq
• [x] #3938 plotting with multiindex
• [x] #3816 map_blocks template
• [x] #3976 in-place replacement error message
• [x] #3975 pint dataset
• [x] #4018 netdf3 dtype coercion
• [ ] #3985 groupby multi-index bug (ref discussion below & in the issue)

From @keewis comment in #4138

I just noticed that the rackcdn.org repository doesn't have matplotlib>=3.2.0, so since about late February we don't test against matplotlib upstream anymore.

ds = xr.Dataset({"a": ("x", np.arange(10)), "b": ("y", np.arange(10))}) ds.rename_dims({"x": "y"}) ValueError: Cannot rename x to y because y already exists. Try using swap_dims instead.

This should be possible since both x and y have the same lengths. Thoughts?

For indexed dimensions, should we check that the indexes are equal?

When I open a notebook, the new HTML repr is "expanded":

I'm running jupyter core : 4.6.3 jupyter-notebook : 6.0.3 qtconsole : 4.7.2 ipython : 7.13.0 ipykernel : 5.2.0 jupyter client : 6.1.2 jupyter lab : 2.1.0 nbconvert : 5.6.1 ipywidgets : 7.5.1 nbformat : 5.0.4 traitlets : 4.3.3 and see this behaviour with both the notebook and jupyterlab interface; on both chrome and firefox (both on linux).

Is anyone else seeing this?

The annotations make this signature basically unreadable to anyone not familiar with them.

Is there a way to hide them in the documentation? Or at least change formatting so it's clearer what the function arguments are?

flake8 is failing on master (https://dev.azure.com/xarray/xarray/_build/results?buildId=2820&view=logs&jobId=a577607c-d99b-546f-eeb4-2341e9a21630&j=a577607c-d99b-546f-eeb4-2341e9a21630&t=7308a173-bf34-5af1-b6d9-30c4d79bebeb) with

``` ========================== Starting Command Output =========================== /bin/bash --noprofile --norc /home/vsts/work/_temp/e6322963-dd1c-4887-ba6a-2aa7ec888f4c.sh ./xarray/backends/memory.py:43:32: E741 ambiguous variable name 'l' ./xarray/backends/common.py:244:32: E741 ambiguous variable name 'l'

[error]Bash exited with code '1'.

Finishing: flake8 lint checks ```

import matplotlib is failing when it imports PIL

```python

E ImportError: dlopen(/usr/local/miniconda/envs/xarray-tests/lib/python3.8/site-packages/PIL/_imaging.cpython-38-darwin.so, 2): Library not loaded: @rpath/libwebp.7.dylib E Referenced from: /usr/local/miniconda/envs/xarray-tests/lib/libtiff.5.dylib E Reason: Incompatible library version: libtiff.5.dylib requires version 9.0.0 or later, but libwebp.7.dylib provides version 8.0.0

/usr/local/miniconda/envs/xarray-tests/lib/python3.8/site-packages/PIL/Image.py:69: ImportError ```

memory consumption errors as usual.

@keewis I remember you had an idea for using pip instead of conda?

It'd be nice to issue a release by the end of March (two months since previous release)

Please edit this list as you wish.

Must-have - [x] #3862 error when assigning to IndexVariable.values - [x] #3840 delete index when deleting coordinate variable - [x] #3874 un-xfail the tests from #3808 (https://github.com/pydata/xarray/issues/3751#issuecomment-599231986) - [x] #3833, #3870 HTML repr with non-str keys fix (since it is now on by default)

Nice to have - [ ] #3816 map_blocks template - [x] #3817 map_blocks new unindexed dimension - [ ] #3818 map_blocks dask args - [ ] #3847 assert_allclose diff - [x] #3826 ellipsis in stack - [x] #3836 where docstring - [ ] some progress on #3868 - [ ] #3594 unit support - [ ] #3871 idxmin, idxmax

https://dev.azure.com/xarray/xarray/_build/results?buildId=2116&view=logs&jobId=2280efed-fda1-53bd-9213-1fa8ec9b4fa8&j=2280efed-fda1-53bd-9213-1fa8ec9b4fa8&t=175181ee-1928-5a6b-f537-168f7a8b7c2d

46 failed tests but they all seem to result from the same TypeError

``` =================================== FAILURES =================================== __ test_sel_date_scalar_nearest[365_day-sel_kwargs0] ___

da = <xarray.DataArray (time: 4)> 0002-02-01 00:00:00], dtype='object') sel_kwargs = {'method': 'nearest'}

@requires_cftime
@pytest.mark.parametrize(
    "sel_kwargs",
    [{"method": "nearest"}, {"method": "nearest", "tolerance": timedelta(days=70)}],
)
def test_sel_date_scalar_nearest(da, date_type, index, sel_kwargs):
    expected = xr.DataArray(2).assign_coords(time=index[1])

  result = da.sel(time=date_type(1, 4, 1), **sel_kwargs)

xarray/tests/test_cftimeindex.py:460:

xarray/core/dataarray.py:1056: in sel ds = self._to_temp_dataset().sel( xarray/core/dataset.py:2056: in sel pos_indexers, new_indexes = remap_label_indexers( xarray/core/coordinates.py:391: in remap_label_indexers pos_indexers, new_indexes = indexing.remap_label_indexers( xarray/core/indexing.py:270: in remap_label_indexers idxr, new_idx = convert_label_indexer(index, label, dim, method, tolerance) xarray/core/indexing.py:189: in convert_label_indexer indexer = index.get_loc( xarray/coding/cftimeindex.py:334: in get_loc return pd.Index.get_loc(self, key, method=method, tolerance=tolerance) /usr/share/miniconda/envs/xarray-tests/lib/python3.8/site-packages/pandas/core/indexes/base.py:2899: in get_loc indexer = self.get_indexer([key], method=method, tolerance=tolerance) /usr/share/miniconda/envs/xarray-tests/lib/python3.8/site-packages/pandas/core/indexes/base.py:2992: in get_indexer indexer = self._get_nearest_indexer(target, limit, tolerance) /usr/share/miniconda/envs/xarray-tests/lib/python3.8/site-packages/pandas/core/indexes/base.py:3076: in _get_nearest_indexer left_distances = np.abs(self[left_indexer] - target) xarray/coding/cftimeindex.py:444: in sub return CFTimeIndex(np.array(self) - other) xarray/coding/cftimeindex.py:248: in new assert_all_valid_date_type(data)

data = TimedeltaIndex(['-59 days'], dtype='timedelta64[ns]', freq=None)

def assert_all_valid_date_type(data):
    import cftime

    if len(data) > 0:
        sample = data[0]
        date_type = type(sample)
        if not isinstance(sample, cftime.datetime):

          raise TypeError(
                "CFTimeIndex requires cftime.datetime "
                "objects. Got object of {}.".format(date_type)
            )

E TypeError: CFTimeIndex requires cftime.datetime objects. Got object of <class 'pandas._libs.tslibs.timedeltas.Timedelta'>.

xarray/coding/cftimeindex.py:206: TypeError ```

Looks like the last run was on Thursday: https://dev.azure.com/xarray/xarray/_build?definitionId=1&_a=summary&view=runs

No tests have been run for PRs #3826 #3836 #3858 and #3807 despite these having been updated recently.

There is a workaround posted at this Azure issue: https://status.dev.azure.com/_event/179641421 but it looks like a fix is coming soon.

TLDR: I think we want A.sel(x=A.x).equals(A) to pass lazily. It doesn't do so currently.

Currently if I do A.sel(x=A.x), this sticks in a getitem call in the dask graph which breaks our lazy array equality optimization. Here's an example

``` python

A = xr.DataArray(np.arange(100), dims="x", coords={"x": np.arange(100)}).chunk({"x": 1}) A.sel(x=A.x).variable.equals(A, equiv=xr.core.duck_array_ops.lazy_array_equiv) None ```

Questions:

1. Where is the best place to do this? In sel or isel? Both? Sticking the following in sel makes the above check return True which is what we want: ``` python if self._indexes: equals = [] for index in indexers: equals.append(indexers[index].to_index().equals(self._indexes[index]))

       if all(equals):
           return self
   

   ```

   This doesn't handle slice objects though so that makes me think we'd want to add something similar to isel too. 2. What is the behaviour we want? A.sel(x=A.x).equals(A) or A.sel(x=A.x) is A? 3. Doing the latter will mean changing _to_temp_dataset and _from_temp_dataset which suggests the constraint A._from_temp_dataset(A._to_temp_dataset()) is A? But this seems too strong to me. Do we only want to lazily satisfy an equals constraint rather than an identical constraint? 4. It seems like we'll want to add such short-circuits in many places (I have not checked all of these): sortby, broadcast, align, reindex (transpose does this now).