1. This moves the region detection code into ZarrStore so we only open the store once.
2. Instead of opening the store as a dataset, construct a pd.Index directly to "auto"-infer the region.

The diff is large mostly because a bunch of code moved from backends/api.py to backends/zarr.py

What happened?

From https://github.com/pydata/xarray/discussions/8951

if d is a MultiIndex-ed dataset with levels (x, y, z), and m is a dataset with a single coord x m.isel(x=d.x) builds a dataset with a MultiIndex with levels (y, z). This seems like it should work.

cc @benbovy

What did you expect to happen?

No response

Minimal Complete Verifiable Example

```Python import pandas as pd, xarray as xr, numpy as np

xr.set_options(use_flox=True)

test = pd.DataFrame() test["x"] = np.arange(100) % 10 test["y"] = np.arange(100) test["z"] = np.arange(100) test["v"] = np.arange(100)

d = xr.Dataset.from_dataframe(test) d = d.set_index(index = ["x", "y", "z"]) print(d)

m = d.groupby("x").mean() print(m)

print(d.xindexes) print(m.isel(x=d.x).xindexes)

xr.align(d, m.isel(x=d.x))

res = d.groupby("x") - m

print(res)

```

<xarray.Dataset> Dimensions: (index: 100) Coordinates: * index (index) object MultiIndex * x (index) int64 0 1 2 3 4 5 6 7 8 9 0 1 2 ... 8 9 0 1 2 3 4 5 6 7 8 9 * y (index) int64 0 1 2 3 4 5 6 7 8 9 ... 90 91 92 93 94 95 96 97 98 99 * z (index) int64 0 1 2 3 4 5 6 7 8 9 ... 90 91 92 93 94 95 96 97 98 99 Data variables: v (index) int64 0 1 2 3 4 5 6 7 8 9 ... 90 91 92 93 94 95 96 97 98 99 <xarray.Dataset> Dimensions: (x: 10) Coordinates: * x (x) int64 0 1 2 3 4 5 6 7 8 9 Data variables: v (x) float64 45.0 46.0 47.0 48.0 49.0 50.0 51.0 52.0 53.0 54.0 Indexes: ┌ index PandasMultiIndex │ x │ y └ z Indexes: ┌ index PandasMultiIndex │ y └ z ValueError...

MVCE confirmation

• [x] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [x] Complete example — the example is self-contained, including all data and the text of any traceback.
• [x] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [x] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [x] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

@benbovy this seems to be the root cause of #8646, the variable name in Dataset._variables does not match IndexVariable.name.

A good number of tests seem to fail though, so not sure if this is a good chck.

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

What happened?

Resetting a MultiIndex dim coordinate preserves the MultiIndex levels as IndexVariables. We should either reset the indexes for the multiindex level variables, or warn asking the users to do so

This seems to be the root cause exposed by https://github.com/pydata/xarray/pull/8809

cc @benbovy

What did you expect to happen?

No response

Minimal Complete Verifiable Example

```Python import numpy as np import xarray as xr

ND DataArray that gets stacked along a multiindex

da = xr.DataArray(np.ones((3, 3)), coords={"dim1": [1, 2, 3], "dim2": [4, 5, 6]}) da = da.stack(feature=["dim1", "dim2"])

Extract just the stacked coordinates for saving in a dataset

ds = xr.Dataset(data_vars={"feature": da.feature}) xr.testing.assertions._assert_internal_invariants(ds.reset_index(["feature", "dim1", "dim2"]), check_default_indexes=False) # succeeds xr.testing.assertions._assert_internal_invariants(ds.reset_index(["feature"]), check_default_indexes=False) # fails, but no warning either ```

What happened?

(xref #8658)

I'm not sure what the expected behaviour is here:

```python import xarray as xr import numpy as np from xarray.testing import _assert_internal_invariants

ds = xr.Dataset() ds.coords["1"] = ("1", np.array([1], dtype=np.uint32)) ds["1_"] = ("1", np.array([1], dtype=np.uint32)) ds = ds.rename_vars({"1": "0"}) ds ```

It looks like this sequence of operations creates a default index

But then ```python from xarray.testing import _assert_internal_invariants

_assert_internal_invariants(ds, check_default_indexes=True) fails with ... File ~/repos/xarray/xarray/testing/assertions.py:301, in _assert_indexes_invariants_checks(indexes, possible_coord_variables, dims, check_default) 299 if check_default: 300 defaults = default_indexes(possible_coord_variables, dims) --> 301 assert indexes.keys() == defaults.keys(), (set(indexes), set(defaults)) 302 assert all(v.equals(defaults[k]) for k, v in indexes.items()), ( 303 indexes, 304 defaults, 305 )

AssertionError: ({'0'}, set()) ```

What happened?

All tests are failing because of an error in create_test_data

from xarray.tests import create_test_data create_test_data()

```

AssertionError Traceback (most recent call last) Cell In[3], line 2 1 from xarray.tests import create_test_data ----> 2 create_test_data()

File ~/repos/xarray/xarray/tests/init.py:329, in create_test_data(seed, add_attrs, dim_sizes) 327 obj.coords["numbers"] = ("dim3", numbers_values) 328 obj.encoding = {"foo": "bar"} --> 329 assert all(var.values.flags.writeable for var in obj.variables.values()) 330 return obj

AssertionError: ```

This fixes the failure locally for me.

cc @max-sixty

Using 255 as fillvalue for int8 arrays will not be allowed any more. Previously this overflowed to -1. Now specify that instead.

On numpy 1.24.4 ```

np.array([255], dtype="i1") DeprecationWarning: NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays. The conversion of 255 to int8 will fail in the future.

array([-1], dtype=int8) ```

Spotted when profiling some write workloads. 1. Speeds up the check for multi-index 2. Speeds up one string encoder by not re-creating variables when not necessary.

@benbovy is there a better way?

What is your issue?

https://docs.xarray.dev/en/stable/getting-started-guide/installing.html#minimum-dependency-versions says that we follow NEP-29, and I think our min versions script also does that.

I propose we follow https://scientific-python.org/specs/spec-0000/

In practice, I think this means we mostly drop Python versions earlier.

• [x] Closes #2157
• [ ] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Could use a close-ish review.

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

xref #8339 xref #8350

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

What is your issue?

As we work through refactoring for NamedArray, it's pretty clear that Xarray will depend pretty closely on many files in namedarray/. For example various utils.py, pycompat.py, *ops.py, formatting.py, formatting_html.py at least. This promises to be quite painful if we did break NamedArray out in to its own repo (particularly around typing, e.g. https://github.com/pydata/xarray/pull/8309)

I propose we use pydata/xarray as a monorepo that serves two packages: NamedArray and Xarray. - We can move as much as is needed to have NamedArray be independent of Xarray, but Xarray will depend quite closely on many utility functions in NamedArray. - We can release both at the same time similar to dask and distributed. - We can re-evaluate if and when NamedArray grows its own community.

• xref #3937
• [ ] Tests added
• [ ] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Initial proof-of-concept. Suggestions to improve this are very welcome.

Here's some convenient testing code ``` python
import xarray as xr

ds = xr.tutorial.open_dataset('rasm').load() month_length = ds.time.dt.days_in_month weights = month_length.groupby('time.season') / month_length.groupby('time.season').sum()

actual = ds.weighted(month_length).groupby("time.season").mean() expected = (ds * weights).groupby('time.season').sum(skipna=False) xr.testing.assert_allclose(actual, expected) ```

I've added info to the repr python ds.weighted(month_length).groupby("time.season") WeightedDatasetGroupBy, grouped over 'season' 4 groups with labels 'DJF', 'JJA', 'MAM', 'SON'. weighted along dimensions: time by 'days_in_month'

I'm opening this issue for discussion really.

I stumbled on autoray (Github) by @jcmgray which provides an abstract interface to a number of array types.

What struck me was the very general lazy compute system. This opens up the possibility of lazy-but-not-dask computation.

Related: https://github.com/pydata/xarray/issues/2298 https://github.com/pydata/xarray/issues/1725 https://github.com/pydata/xarray/issues/5081

This adds a benchmark of HEAD vs the latest tag on main.

xref #7730

Is your feature request related to a problem?

See https://github.com/pydata/xarray/pull/6213/files#r795716713

We check whether the by variable for groupby is unique and monotonically increasing. But the fast path would also apply to unique and monotonically decreasing variables.

Describe the solution you'd like

Update the condition to is_monotonic_increasing or is_monotonic_decreasing and add a test.

Describe alternatives you've considered

No response

Additional context

No response

Is your feature request related to a problem?

Here are two datasets, in one a is a data_var, in the other a is a coordinate variable. The following fails ``` python import xarray as xr

ds1 = xr.Dataset({"a": ('x', [1, 2, 3])}) ds2 = ds1.set_coords("a") ds2.update(ds1) with 649 ambiguous_coords = coord_names.intersection(noncoord_names) 650 if ambiguous_coords: --> 651 raise MergeError( 652 "unable to determine if these variables should be " 653 f"coordinates or not in the merged result: {ambiguous_coords}" 654 ) 656 attrs = merge_attrs( 657 [var.attrs for var in coerced if isinstance(var, (Dataset, DataArray))], 658 combine_attrs, 659 ) 661 return _MergeResult(variables, coord_names, dims, out_indexes, attrs)

MergeError: unable to determine if these variables should be coordinates or not in the merged result: {'a'} ```

Describe the solution you'd like

I think we should replace this error with a warning and arbitrarily choose to either convert a to a coordinate variable or a data variable.

Describe alternatives you've considered

No response

Additional context

No response

• [x] Closes #6505
• [x] Tests added

What happened?

Seen in #4972

``` python import xarray as xr import numpy as np

array = np.array([0. +0.j, 0.+np.nan * 1j], dtype=np.complex64) var = xr.Variable("x", array) print(var.mean().data) print(array.mean()) ```

0j (nan+nanj)

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

• [x] Closes #6505
• [x] Tests added

What is your issue?

We have some stale issue but not that many.

Can we restrict the bot to only issues that are untagged, or tagged as "usage question" or are not assigned to a "project" instead? This might reduce a lot of the noise.

xref #6293

• [x] Closes #6430
• [x] Tests added

What happened?

``` python import numpy as np import xarray as xr

ds = xr.Dataset( {"foo": (("x", "y", "z"), np.ones((3, 4, 2)))}, {"x": ["a", "b", "c"], "y": [1, 2, 3, 4]}, ) expected = ds.sum("z") stacked = ds.stack(space=["x", "y"])

broadcasted, _ = xr.broadcast(stacked, stacked.space)

stacked.sum("z").unstack("space") # works broadcasted.sum("z").unstack("space") # error ```

```

ValueError Traceback (most recent call last) Input In [13], in <module> 10 broadcasted, _ = xr.broadcast(stacked, stacked.space) 11 stacked.sum("z").unstack("space") ---> 12 broadcasted.sum("z").unstack("space")

File ~/work/python/xarray/xarray/core/dataset.py:4332, in Dataset.unstack(self, dim, fill_value, sparse) 4330 non_multi_dims = set(dims) - set(stacked_indexes) 4331 if non_multi_dims: -> 4332 raise ValueError( 4333 "cannot unstack dimensions that do not " 4334 f"have exactly one multi-index: {tuple(non_multi_dims)}" 4335 ) 4337 result = self.copy(deep=False) 4339 # we want to avoid allocating an object-dtype ndarray for a MultiIndex, 4340 # so we can't just access self.variables[v].data for every variable. 4341 # We only check the non-index variables. 4342 # https://github.com/pydata/xarray/issues/5902

ValueError: cannot unstack dimensions that do not have exactly one multi-index: ('space',) ```

What did you expect to happen?

This should work.

Minimal Complete Verifiable Example

No response

Relevant log output

No response

Anything else we need to know?

No response

Environment

xarray main after the flexible indexes refactor

What happened?

``` python import xarray as xr

xr.where(xr.DataArray([1, 2, 3]) > 0, 1, 0) ```

fails with

`` 1809 if keep_attrs is True: 1810 # keep the attributes of x, the second parameter, by default to 1811 # be consistent with thewheremethod ofDataArrayandDataset` -> 1812 keep_attrs = lambda attrs, context: attrs[1] 1814 # alignment for three arguments is complicated, so don't support it yet 1815 return apply_ufunc( 1816 duck_array_ops.where, 1817 cond, (...) 1823 keep_attrs=keep_attrs, 1824 )

IndexError: list index out of range ```

The workaround is to pass keep_attrs=False

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

Relevant log output

No response

Anything else we need to know?

No response

Environment

xarray 2022.3.0

version rasterio 1.1.1 py36h900e953_0 conda-forge

``` =================================== FAILURES =================================== ___ TestRasterio.testrasterio_vrt ____

self = <xarray.tests.test_backends.TestRasterio object at 0x7fc8355c8f60>

def test_rasterio_vrt(self):
    import rasterio

    # tmp_file default crs is UTM: CRS({'init': 'epsg:32618'}
    with create_tmp_geotiff() as (tmp_file, expected):
        with rasterio.open(tmp_file) as src:
            with rasterio.vrt.WarpedVRT(src, crs="epsg:4326") as vrt:
                expected_shape = (vrt.width, vrt.height)
                expected_crs = vrt.crs
                expected_res = vrt.res
                # Value of single pixel in center of image
                lon, lat = vrt.xy(vrt.width // 2, vrt.height // 2)

              expected_val = next(vrt.sample([(lon, lat)]))

xarray/tests/test_backends.py:3966:

/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/rasterio/sample.py:43: in sample_gen data = read(indexes, window=window, masked=masked, boundless=True)

??? E ValueError: WarpedVRT does not permit boundless reads

rasterio/_warp.pyx:978: ValueError ```

By switching to numpy_groupies we are vectorizing our groupby reductions. I think we can do the same for groupby's binary ops.

Here's an example array ``` python import numpy as np import xarray as xr

%load_ext memory_profiler

N = 4 * 2000 da = xr.DataArray( np.random.random((N, N)), dims=("x", "y"), coords={"labels": ("x", np.repeat(["a", "b", "c", "d", "e", "f", "g", "h"], repeats=N//8))}, ) ```

Consider this "anomaly" calculation, anomaly defined relative to the group mean

``` python def anom_current(da): grouped = da.groupby("labels") mean = grouped.mean() anom = grouped - mean return anom

```

With this approach, we loop over each group and apply the binary operation: https://github.com/pydata/xarray/blob/a1635d324753588e353e4e747f6058936fa8cf1e/xarray/core/computation.py#L502-L525

This saves some memory, but becomes slow for large number of groups.

We could instead do def anom_vectorized(da): mean = da.groupby("labels").mean() mean_expanded = mean.sel(labels=da.labels) anom = da - mean_expanded return anom

Now we are faster, but construct an extra array as big as the original array (I think this is an OK tradeoff). ``` %timeit anom_current(da)

1.4 s ± 20.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit anom_vectorized(da)

937 ms ± 5.26 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```

(I haven't experimented with dask yet, so the following is just a theory).

I think the real benefit comes with dask. Depending on where the groups are located relative to chunking, we could end up creating a lot of tiny chunks by splitting up existing chunks. With the vectorized approach we can do better.

Ideally we would reindex the "mean" dask array with a numpy-array-of-repeated-ints such that the chunking of mean_expanded exactly matches the chunking of da along the grouped dimension.

~In practice, dask.array.take doesn't allow specifying "output chunks" so we'd end up chunking "mean_expanded" based on dask's automatic heuristics, and then rechunking again for the binary operation.~

Thoughts?

cc @rabernat

What happened?

After vectorized indexing a DataArray with dim xby a DataArray z, we get a DataArray with dim z and x as non-dim coordinate. But x is still an IndexVariable, not a normal variable.

What did you expect to happen?

x should be a normal variable.

Minimal Complete Verifiable Example

```python import xarray as xr xr.set_options(display_style="text")

da = xr.DataArray([1, 2, 3], dims="x", coords={"x": [0, 1, 2]}) idxr = xr.DataArray([1], dims="z", name="x", coords={"z": ("z", ["a"])}) da.sel(x=idxr) ```

<xarray.DataArray (z: 1)> array([2]) Coordinates: x (z) int64 1 * z (z) <U1 'a'

x is a non-dim coordinate but is backed by a IndexVariable with the wrong name! python da.sel(x=idxr).x.variable

<xarray.IndexVariable 'z' (z: 1)> array([1])

Relevant log output

No response

Anything else we need to know?

No response

Environment

xarray main but this bug was present prior to the explicit indexes refactor.

• [x] Closes #2060, closes #2575, xref #1614
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Makes some error messages clearer

Some dask array cleanups

1. switch to using dask.array.map_blocks instead of Array.map_blocks (duck dask array compatibility)
2. Stop vendoring meta_from_array and median

• [x] Closes #3734
• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Whats-new fixes for the release.

Feel free to push to this branch with more improvements.

This reverts commit d6ee8caa84b27d4635ec3384b1a06ef4ddf2d998.

see https://github.com/pydata/xarray/pull/3285

I found it useful to ignore big reformatting commits in git blame. See https://www.michaelheap.com/git-ignore-rev/

it's opt-in using a command-line flag or you can set git config --global blame.ignoreRevsFile .git-blame-ignore-revs

Thoughts on adding it to the repo? If so, are there more commits we can add?

xref https://github.com/pydata/xarray/pull/5636/files#r679823542

avoid len(index) in two places.

Continuing discussion form #3491.

Pandas has added attrs to their objects. We should decide on what to do with them in the DataArray constructor. Many tests fail if we don't handle this case explicitly.

@dcherian:

Not sure what we want to do about these attributes in the long term. One option would be to pop the name attribute, assign to DataArray.name and keep the rest as DataArray.attrs? But what if name clashes with the provided name?

@max-sixty:

Agree! I think we could prioritize the supplied name above that in attrs. Another option would be raising an error if both were supplied.

• xref #5409

Just moves the tests out, in preparation for numpy_groupies work

There are a few tests for .assign and .fillna (for e.g.) still present in test_dataset

The DataArray tests are not a simple copy and paste :(

The RTD build is failing on all PRs with

Sphinx parallel build error: nbsphinx.NotebookError: UndefinedError in examples/ERA5-GRIB-example.ipynb: 'nbformat.notebooknode.NotebookNode object' has no attribute 'tags'

What happened: The formatting of section headings "for users", "community" etc. has changed: https://xarray.pydata.org/en/latest/

Just as a reminder.

Right now all rolling operations with dask arrays use .construct().reduce().

Downstream issue: https://github.com/xarray-contrib/cf-xarray/issues/57 Example: no reprs displayed in https://cf-xarray.readthedocs.io/en/latest/examples/introduction.html

@benbovy's diagnosis:

It looks like bootstrap 4 (used by sphinx-book-theme) forces all html elements with hidden attributes to be actually hidden (source), so the hack in pydata/xarray#4053 does not work here (the result is even worse). I guess that a workaround would be to add some custom CSS such as .xr-wrap { display: block !important }, assuming that custom CSS is loaded after Bootstrap's CSS. Not ideal, though, it looks like a hack on top of another hack.

• [ ] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Significantly speeds up _construct_dataarray by iterating over ._coord_names instead of .coords. This avoids unnecessarily constructing a DatasetCoordinates object and massively speeds up repr construction for datasets with large numbers of variables.

Construct a 2000 variable dataset ```python import numpy as np import xarray as xr

a = np.arange(0, 2000) b = np.core.defchararray.add("long_variable_name", a.astype(str)) coords = dict(time=np.array([0, 1])) data_vars = dict() for v in b: data_vars[v] = xr.DataArray( name=v, data=np.array([3, 4]), dims=["time"], coords=coords ) ds0 = xr.Dataset(data_vars) ```

Before: ``` %timeit ds0['long_variable_name1999'] %timeit ds0.repr()

1.33 ms ± 23 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) 2.66 s ± 52.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) ```

After:

``` %timeit ds0['long_variable_name1999'] %timeit ds0.repr()

10.5 µs ± 203 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) 84.2 ms ± 1.28 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) ```

Is your feature request related to a problem? Please describe. I have datasets with lots of non-dim coord variables. Its annoying to search through and look at the dimension coordinates to get an idea of what subset of data I am looking at.

Describe the solution you'd like I think we should show dimension coordinate variables at the top of the coordinates repr.

Example code

python ds = xr.Dataset() ds.coords["as"] = 10 ds["var"] = xr.DataArray(np.ones((10,)), dims="x", coords={"x": np.arange(10)}) ds

Related #4409

See https://readthedocs.org/projects/xray/builds/

There's no useful information in the logs AFAICT: e.g. https://readthedocs.org/projects/xray/builds/11504571/

flake8 is failing on master (https://dev.azure.com/xarray/xarray/_build/results?buildId=2820&view=logs&jobId=a577607c-d99b-546f-eeb4-2341e9a21630&j=a577607c-d99b-546f-eeb4-2341e9a21630&t=7308a173-bf34-5af1-b6d9-30c4d79bebeb) with

``` ========================== Starting Command Output =========================== /bin/bash --noprofile --norc /home/vsts/work/_temp/e6322963-dd1c-4887-ba6a-2aa7ec888f4c.sh ./xarray/backends/memory.py:43:32: E741 ambiguous variable name 'l' ./xarray/backends/common.py:244:32: E741 ambiguous variable name 'l'

[error]Bash exited with code '1'.

Finishing: flake8 lint checks ```

Make some zarr error messages more helpful.

• [x] Closes #3524
• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Small change that makes map_blocks apply functions that add new unindexed dimensions.

Looks like the last run was on Thursday: https://dev.azure.com/xarray/xarray/_build?definitionId=1&_a=summary&view=runs

No tests have been run for PRs #3826 #3836 #3858 and #3807 despite these having been updated recently.

There is a workaround posted at this Azure issue: https://status.dev.azure.com/_event/179641421 but it looks like a fix is coming soon.

• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

da1 = xr.DataArray([1, 2, 3], dims="x", coords={"x": [1, 2, 3], "y": 1}) da2 = xr.DataArray([4, 5, 6], dims="x", coords={"x": [4, 5, 6]}) xr.concat([da1, da2], dim="x") This use case is quite common since you can get da1 from something like bigger_da1.sel(y=1)

On master this raises an uninformative KeyError because 'y' is not present in all datasets. This is because coords="different" by default which means that we are checking for equality. However coords='different'(and the equality check) is meaningless when the variable is only present in one of the objects to be concatenated.

This PR skips equality checking when a variable is only present in one dataset and raises a nicer error message when it is present in more than one but not all datasets.

• [x] Closes #3603

Adds support for dask arrays.

• [x] Closes #3326
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

happy to make changes!

• [x] Closes #3402
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

It would be good to link to https://docs.dask.org/en/latest/array-chunks.html in https://xarray.pydata.org/en/stable/dask.html#chunking-and-performance

• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Thoughts on adding something like this?

• [x] Closes #3280

This undoes the auto_combine deprecation until we figure out the best way to proceed.

(I am not very familiar with auto_combine so someone else should look over this. The tests all pass though...)

• [x] Closes #2580

Use open_dataset.load() instead of load_dataset()

• [x] Closes #2208 (remove if there is no corresponding issue, which should only be the case for minor changes)

This fixes it for me locally. The height was forced to be 12in, while width is 100%.

I'm not sure why this was added in the first place.

• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

The current version uses pd.isnull to remove invalid values from input data when making a colorbar. pd.isnull([np.inf]) is False which means _determine_cmap_params returns Inf for colorbar limits which screws everything up. This PR changes pd.isnull to np.isfinite.

• [x] Closes #2871
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Closes #2239

Fixes a broken doc image: http://xarray.pydata.org/en/stable/plotting.html#id4

The tests passed previously because there was no metadata associated with the test DataArray. I've assigned some now, that should be good enough.

• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [x] Closes #2610

Quickfixes to make things work locally.

Sample error from https://travis-ci.org/pydata/xarray/jobs/466431752

``` _____ test_dataarray_with_dask_coords ______ def test_dataarray_with_dask_coords(): import toolz x = xr.Variable('x', da.arange(8, chunks=(4,))) y = xr.Variable('y', da.arange(8, chunks=(4,)) * 2) data = da.random.random((8, 8), chunks=(4, 4)) + 1 array = xr.DataArray(data, dims=['x', 'y']) array.coords['xx'] = x array.coords['yy'] = y

    assert dict(array.__dask_graph__()) == toolz.merge(data.__dask_graph__(),
                                                       x.__dask_graph__(),
                                                       y.__dask_graph__())

  (array2,) = dask.compute(array)

xarray/tests/test_dask.py:824:

../../../miniconda/envs/test_env/lib/python3.6/site-packages/dask/base.py:395: in compute dsk = collections_to_dsk(collections, optimize_graph, *kwargs) ../../../miniconda/envs/test_env/lib/python3.6/site-packages/dask/base.py:187: in collections_to_dsk for opt, val in groups.items()} ../../../miniconda/envs/test_env/lib/python3.6/site-packages/dask/base.py:187: in <dictcomp> for opt, val in groups.items()} ../../../miniconda/envs/test_env/lib/python3.6/site-packages/dask/base.py:212: in _extract_graph_and_keys graph = merge(graphs)

dicts = <dask.sharedict.ShareDict object at 0x7f307d29a128>, kwargs = {} factory = <class 'dict'>, rv = {} d = ('arange-36f53ab1e6153a63bbf7f4f8ff56693c', 0) def merge(dicts, *kwargs): """ Merge a collection of dictionaries

    >>> merge({1: 'one'}, {2: 'two'})
    {1: 'one', 2: 'two'}

    Later dictionaries have precedence

    >>> merge({1: 2, 3: 4}, {3: 3, 4: 4})
    {1: 2, 3: 3, 4: 4}

    See Also:
        merge_with
    """
    if len(dicts) == 1 and not isinstance(dicts[0], dict):
        dicts = dicts[0]
    factory = _get_factory(merge, kwargs)

    rv = factory()
    for d in dicts:

      rv.update(d)

E ValueError: dictionary update sequence element #0 has length 39; 2 is required ../../../miniconda/envs/test_env/lib/python3.6/site-packages/toolz/dicttoolz.py:39: ValueError ```

• [x] Closes #2512 (remove if there is no corresponding issue, which should only be the case for minor changes)
• [x] Tests added (for all bug fixes or enhancements)

• [x] Closes #1504, closes #1717, closes #1735
• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

1735 is stalled, so I jumped in.

I've added an error if cbar_ax is provided as an option for FacetGrid. Don't think it's really needed.

• [x] Closes #2381 (remove if there is no corresponding issue, which should only be the case for minor changes)
• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

Rotate x-axis dateticks by default, just as for plot.line()

• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Passes git diff upstream/master **/*py | flake8 --diff (remove if you did not edit any Python files)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

I like @fujiisoup's weighted mean demo in this stack overflow example: https://stackoverflow.com/questions/48510784/xarray-rolling-mean-with-weights

I thought it'd be a useful addition to the docs on rolling.