What happened?

(xref #8658)

I'm not sure what the expected behaviour is here:

```python import xarray as xr import numpy as np from xarray.testing import _assert_internal_invariants

ds = xr.Dataset() ds.coords["1"] = ("1", np.array([1], dtype=np.uint32)) ds["1_"] = ("1", np.array([1], dtype=np.uint32)) ds = ds.rename_vars({"1": "0"}) ds ```

It looks like this sequence of operations creates a default index

But then ```python from xarray.testing import _assert_internal_invariants

_assert_internal_invariants(ds, check_default_indexes=True) fails with ... File ~/repos/xarray/xarray/testing/assertions.py:301, in _assert_indexes_invariants_checks(indexes, possible_coord_variables, dims, check_default) 299 if check_default: 300 defaults = default_indexes(possible_coord_variables, dims) --> 301 assert indexes.keys() == defaults.keys(), (set(indexes), set(defaults)) 302 assert all(v.equals(defaults[k]) for k, v in indexes.items()), ( 303 indexes, 304 defaults, 305 )

AssertionError: ({'0'}, set()) ```

What happened?

All tests are failing because of an error in create_test_data

from xarray.tests import create_test_data create_test_data()

```

AssertionError Traceback (most recent call last) Cell In[3], line 2 1 from xarray.tests import create_test_data ----> 2 create_test_data()

File ~/repos/xarray/xarray/tests/init.py:329, in create_test_data(seed, add_attrs, dim_sizes) 327 obj.coords["numbers"] = ("dim3", numbers_values) 328 obj.encoding = {"foo": "bar"} --> 329 assert all(var.values.flags.writeable for var in obj.variables.values()) 330 return obj

AssertionError: ```

This fixes the failure locally for me.

cc @max-sixty

Using 255 as fillvalue for int8 arrays will not be allowed any more. Previously this overflowed to -1. Now specify that instead.

On numpy 1.24.4 ```

np.array([255], dtype="i1") DeprecationWarning: NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays. The conversion of 255 to int8 will fail in the future.

array([-1], dtype=int8) ```

Spotted when profiling some write workloads. 1. Speeds up the check for multi-index 2. Speeds up one string encoder by not re-creating variables when not necessary.

@benbovy is there a better way?

• [x] Closes #2157
• [ ] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Could use a close-ish review.

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

xref #8339 xref #8350

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

This adds a benchmark of HEAD vs the latest tag on main.

xref #7730

• [x] Closes #6505
• [x] Tests added

What happened?

Seen in #4972

``` python import xarray as xr import numpy as np

array = np.array([0. +0.j, 0.+np.nan * 1j], dtype=np.complex64) var = xr.Variable("x", array) print(var.mean().data) print(array.mean()) ```

0j (nan+nanj)

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

• [x] Closes #6505
• [x] Tests added

What is your issue?

We have some stale issue but not that many.

Can we restrict the bot to only issues that are untagged, or tagged as "usage question" or are not assigned to a "project" instead? This might reduce a lot of the noise.

xref #6293

• [x] Closes #6430
• [x] Tests added

What happened?

``` python import numpy as np import xarray as xr

ds = xr.Dataset( {"foo": (("x", "y", "z"), np.ones((3, 4, 2)))}, {"x": ["a", "b", "c"], "y": [1, 2, 3, 4]}, ) expected = ds.sum("z") stacked = ds.stack(space=["x", "y"])

broadcasted, _ = xr.broadcast(stacked, stacked.space)

stacked.sum("z").unstack("space") # works broadcasted.sum("z").unstack("space") # error ```

```

ValueError Traceback (most recent call last) Input In [13], in <module> 10 broadcasted, _ = xr.broadcast(stacked, stacked.space) 11 stacked.sum("z").unstack("space") ---> 12 broadcasted.sum("z").unstack("space")

File ~/work/python/xarray/xarray/core/dataset.py:4332, in Dataset.unstack(self, dim, fill_value, sparse) 4330 non_multi_dims = set(dims) - set(stacked_indexes) 4331 if non_multi_dims: -> 4332 raise ValueError( 4333 "cannot unstack dimensions that do not " 4334 f"have exactly one multi-index: {tuple(non_multi_dims)}" 4335 ) 4337 result = self.copy(deep=False) 4339 # we want to avoid allocating an object-dtype ndarray for a MultiIndex, 4340 # so we can't just access self.variables[v].data for every variable. 4341 # We only check the non-index variables. 4342 # https://github.com/pydata/xarray/issues/5902

ValueError: cannot unstack dimensions that do not have exactly one multi-index: ('space',) ```

What did you expect to happen?

This should work.

Minimal Complete Verifiable Example

No response

Relevant log output

No response

Anything else we need to know?

No response

Environment

xarray main after the flexible indexes refactor

What happened?

``` python import xarray as xr

xr.where(xr.DataArray([1, 2, 3]) > 0, 1, 0) ```

fails with

`` 1809 if keep_attrs is True: 1810 # keep the attributes of x, the second parameter, by default to 1811 # be consistent with thewheremethod ofDataArrayandDataset` -> 1812 keep_attrs = lambda attrs, context: attrs[1] 1814 # alignment for three arguments is complicated, so don't support it yet 1815 return apply_ufunc( 1816 duck_array_ops.where, 1817 cond, (...) 1823 keep_attrs=keep_attrs, 1824 )

IndexError: list index out of range ```

The workaround is to pass keep_attrs=False

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

Relevant log output

No response

Anything else we need to know?

No response

Environment

xarray 2022.3.0

version rasterio 1.1.1 py36h900e953_0 conda-forge

``` =================================== FAILURES =================================== ___ TestRasterio.testrasterio_vrt ____

self = <xarray.tests.test_backends.TestRasterio object at 0x7fc8355c8f60>

def test_rasterio_vrt(self):
    import rasterio

    # tmp_file default crs is UTM: CRS({'init': 'epsg:32618'}
    with create_tmp_geotiff() as (tmp_file, expected):
        with rasterio.open(tmp_file) as src:
            with rasterio.vrt.WarpedVRT(src, crs="epsg:4326") as vrt:
                expected_shape = (vrt.width, vrt.height)
                expected_crs = vrt.crs
                expected_res = vrt.res
                # Value of single pixel in center of image
                lon, lat = vrt.xy(vrt.width // 2, vrt.height // 2)

              expected_val = next(vrt.sample([(lon, lat)]))

xarray/tests/test_backends.py:3966:

/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/rasterio/sample.py:43: in sample_gen data = read(indexes, window=window, masked=masked, boundless=True)

??? E ValueError: WarpedVRT does not permit boundless reads

rasterio/_warp.pyx:978: ValueError ```

By switching to numpy_groupies we are vectorizing our groupby reductions. I think we can do the same for groupby's binary ops.

Here's an example array ``` python import numpy as np import xarray as xr

%load_ext memory_profiler

N = 4 * 2000 da = xr.DataArray( np.random.random((N, N)), dims=("x", "y"), coords={"labels": ("x", np.repeat(["a", "b", "c", "d", "e", "f", "g", "h"], repeats=N//8))}, ) ```

Consider this "anomaly" calculation, anomaly defined relative to the group mean

``` python def anom_current(da): grouped = da.groupby("labels") mean = grouped.mean() anom = grouped - mean return anom

```

With this approach, we loop over each group and apply the binary operation: https://github.com/pydata/xarray/blob/a1635d324753588e353e4e747f6058936fa8cf1e/xarray/core/computation.py#L502-L525

This saves some memory, but becomes slow for large number of groups.

We could instead do def anom_vectorized(da): mean = da.groupby("labels").mean() mean_expanded = mean.sel(labels=da.labels) anom = da - mean_expanded return anom

Now we are faster, but construct an extra array as big as the original array (I think this is an OK tradeoff). ``` %timeit anom_current(da)

1.4 s ± 20.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit anom_vectorized(da)

937 ms ± 5.26 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```

(I haven't experimented with dask yet, so the following is just a theory).

I think the real benefit comes with dask. Depending on where the groups are located relative to chunking, we could end up creating a lot of tiny chunks by splitting up existing chunks. With the vectorized approach we can do better.

Ideally we would reindex the "mean" dask array with a numpy-array-of-repeated-ints such that the chunking of mean_expanded exactly matches the chunking of da along the grouped dimension.

~In practice, dask.array.take doesn't allow specifying "output chunks" so we'd end up chunking "mean_expanded" based on dask's automatic heuristics, and then rechunking again for the binary operation.~

Thoughts?

cc @rabernat

What happened?

After vectorized indexing a DataArray with dim xby a DataArray z, we get a DataArray with dim z and x as non-dim coordinate. But x is still an IndexVariable, not a normal variable.

What did you expect to happen?

x should be a normal variable.

Minimal Complete Verifiable Example

```python import xarray as xr xr.set_options(display_style="text")

da = xr.DataArray([1, 2, 3], dims="x", coords={"x": [0, 1, 2]}) idxr = xr.DataArray([1], dims="z", name="x", coords={"z": ("z", ["a"])}) da.sel(x=idxr) ```

<xarray.DataArray (z: 1)> array([2]) Coordinates: x (z) int64 1 * z (z) <U1 'a'

x is a non-dim coordinate but is backed by a IndexVariable with the wrong name! python da.sel(x=idxr).x.variable

<xarray.IndexVariable 'z' (z: 1)> array([1])

Relevant log output

No response

Anything else we need to know?

No response

Environment

xarray main but this bug was present prior to the explicit indexes refactor.

• [x] Closes #2060, closes #2575, xref #1614
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Makes some error messages clearer

Some dask array cleanups

1. switch to using dask.array.map_blocks instead of Array.map_blocks (duck dask array compatibility)
2. Stop vendoring meta_from_array and median

• [x] Closes #3734
• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Whats-new fixes for the release.

Feel free to push to this branch with more improvements.

This reverts commit d6ee8caa84b27d4635ec3384b1a06ef4ddf2d998.

I found it useful to ignore big reformatting commits in git blame. See https://www.michaelheap.com/git-ignore-rev/

it's opt-in using a command-line flag or you can set git config --global blame.ignoreRevsFile .git-blame-ignore-revs

Thoughts on adding it to the repo? If so, are there more commits we can add?

xref https://github.com/pydata/xarray/pull/5636/files#r679823542

avoid len(index) in two places.

• xref #5409

Just moves the tests out, in preparation for numpy_groupies work

There are a few tests for .assign and .fillna (for e.g.) still present in test_dataset

The DataArray tests are not a simple copy and paste :(

The RTD build is failing on all PRs with

Sphinx parallel build error: nbsphinx.NotebookError: UndefinedError in examples/ERA5-GRIB-example.ipynb: 'nbformat.notebooknode.NotebookNode object' has no attribute 'tags'

What happened: The formatting of section headings "for users", "community" etc. has changed: https://xarray.pydata.org/en/latest/

Just as a reminder.

Right now all rolling operations with dask arrays use .construct().reduce().

Downstream issue: https://github.com/xarray-contrib/cf-xarray/issues/57 Example: no reprs displayed in https://cf-xarray.readthedocs.io/en/latest/examples/introduction.html

@benbovy's diagnosis:

It looks like bootstrap 4 (used by sphinx-book-theme) forces all html elements with hidden attributes to be actually hidden (source), so the hack in pydata/xarray#4053 does not work here (the result is even worse). I guess that a workaround would be to add some custom CSS such as .xr-wrap { display: block !important }, assuming that custom CSS is loaded after Bootstrap's CSS. Not ideal, though, it looks like a hack on top of another hack.

• [ ] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Significantly speeds up _construct_dataarray by iterating over ._coord_names instead of .coords. This avoids unnecessarily constructing a DatasetCoordinates object and massively speeds up repr construction for datasets with large numbers of variables.

Construct a 2000 variable dataset ```python import numpy as np import xarray as xr

a = np.arange(0, 2000) b = np.core.defchararray.add("long_variable_name", a.astype(str)) coords = dict(time=np.array([0, 1])) data_vars = dict() for v in b: data_vars[v] = xr.DataArray( name=v, data=np.array([3, 4]), dims=["time"], coords=coords ) ds0 = xr.Dataset(data_vars) ```

Before: ``` %timeit ds0['long_variable_name1999'] %timeit ds0.repr()

1.33 ms ± 23 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) 2.66 s ± 52.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) ```

After:

``` %timeit ds0['long_variable_name1999'] %timeit ds0.repr()

10.5 µs ± 203 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) 84.2 ms ± 1.28 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) ```

Is your feature request related to a problem? Please describe. I have datasets with lots of non-dim coord variables. Its annoying to search through and look at the dimension coordinates to get an idea of what subset of data I am looking at.

Describe the solution you'd like I think we should show dimension coordinate variables at the top of the coordinates repr.

Example code

python ds = xr.Dataset() ds.coords["as"] = 10 ds["var"] = xr.DataArray(np.ones((10,)), dims="x", coords={"x": np.arange(10)}) ds

Related #4409

See https://readthedocs.org/projects/xray/builds/

There's no useful information in the logs AFAICT: e.g. https://readthedocs.org/projects/xray/builds/11504571/

flake8 is failing on master (https://dev.azure.com/xarray/xarray/_build/results?buildId=2820&view=logs&jobId=a577607c-d99b-546f-eeb4-2341e9a21630&j=a577607c-d99b-546f-eeb4-2341e9a21630&t=7308a173-bf34-5af1-b6d9-30c4d79bebeb) with

``` ========================== Starting Command Output =========================== /bin/bash --noprofile --norc /home/vsts/work/_temp/e6322963-dd1c-4887-ba6a-2aa7ec888f4c.sh ./xarray/backends/memory.py:43:32: E741 ambiguous variable name 'l' ./xarray/backends/common.py:244:32: E741 ambiguous variable name 'l'

[error]Bash exited with code '1'.

Finishing: flake8 lint checks ```

Make some zarr error messages more helpful.

• [x] Closes #3524
• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Small change that makes map_blocks apply functions that add new unindexed dimensions.

Looks like the last run was on Thursday: https://dev.azure.com/xarray/xarray/_build?definitionId=1&_a=summary&view=runs

No tests have been run for PRs #3826 #3836 #3858 and #3807 despite these having been updated recently.

There is a workaround posted at this Azure issue: https://status.dev.azure.com/_event/179641421 but it looks like a fix is coming soon.

• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

da1 = xr.DataArray([1, 2, 3], dims="x", coords={"x": [1, 2, 3], "y": 1}) da2 = xr.DataArray([4, 5, 6], dims="x", coords={"x": [4, 5, 6]}) xr.concat([da1, da2], dim="x") This use case is quite common since you can get da1 from something like bigger_da1.sel(y=1)

On master this raises an uninformative KeyError because 'y' is not present in all datasets. This is because coords="different" by default which means that we are checking for equality. However coords='different'(and the equality check) is meaningless when the variable is only present in one of the objects to be concatenated.

This PR skips equality checking when a variable is only present in one dataset and raises a nicer error message when it is present in more than one but not all datasets.

• [x] Closes #3603

Adds support for dask arrays.

• [x] Closes #3326
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

happy to make changes!

• [x] Closes #3402
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

It would be good to link to https://docs.dask.org/en/latest/array-chunks.html in https://xarray.pydata.org/en/stable/dask.html#chunking-and-performance

• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Thoughts on adding something like this?

• [x] Closes #3280

This undoes the auto_combine deprecation until we figure out the best way to proceed.

(I am not very familiar with auto_combine so someone else should look over this. The tests all pass though...)

• [x] Closes #2580

Use open_dataset.load() instead of load_dataset()

• [x] Closes #2208 (remove if there is no corresponding issue, which should only be the case for minor changes)

This fixes it for me locally. The height was forced to be 12in, while width is 100%.

I'm not sure why this was added in the first place.

• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

The current version uses pd.isnull to remove invalid values from input data when making a colorbar. pd.isnull([np.inf]) is False which means _determine_cmap_params returns Inf for colorbar limits which screws everything up. This PR changes pd.isnull to np.isfinite.

• [x] Closes #2871
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Closes #2239

Fixes a broken doc image: http://xarray.pydata.org/en/stable/plotting.html#id4

The tests passed previously because there was no metadata associated with the test DataArray. I've assigned some now, that should be good enough.

• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [x] Closes #2610

Quickfixes to make things work locally.

Sample error from https://travis-ci.org/pydata/xarray/jobs/466431752

``` _____ test_dataarray_with_dask_coords ______ def test_dataarray_with_dask_coords(): import toolz x = xr.Variable('x', da.arange(8, chunks=(4,))) y = xr.Variable('y', da.arange(8, chunks=(4,)) * 2) data = da.random.random((8, 8), chunks=(4, 4)) + 1 array = xr.DataArray(data, dims=['x', 'y']) array.coords['xx'] = x array.coords['yy'] = y

    assert dict(array.__dask_graph__()) == toolz.merge(data.__dask_graph__(),
                                                       x.__dask_graph__(),
                                                       y.__dask_graph__())

  (array2,) = dask.compute(array)

xarray/tests/test_dask.py:824:

../../../miniconda/envs/test_env/lib/python3.6/site-packages/dask/base.py:395: in compute dsk = collections_to_dsk(collections, optimize_graph, *kwargs) ../../../miniconda/envs/test_env/lib/python3.6/site-packages/dask/base.py:187: in collections_to_dsk for opt, val in groups.items()} ../../../miniconda/envs/test_env/lib/python3.6/site-packages/dask/base.py:187: in <dictcomp> for opt, val in groups.items()} ../../../miniconda/envs/test_env/lib/python3.6/site-packages/dask/base.py:212: in _extract_graph_and_keys graph = merge(graphs)

dicts = <dask.sharedict.ShareDict object at 0x7f307d29a128>, kwargs = {} factory = <class 'dict'>, rv = {} d = ('arange-36f53ab1e6153a63bbf7f4f8ff56693c', 0) def merge(dicts, *kwargs): """ Merge a collection of dictionaries

    >>> merge({1: 'one'}, {2: 'two'})
    {1: 'one', 2: 'two'}

    Later dictionaries have precedence

    >>> merge({1: 2, 3: 4}, {3: 3, 4: 4})
    {1: 2, 3: 3, 4: 4}

    See Also:
        merge_with
    """
    if len(dicts) == 1 and not isinstance(dicts[0], dict):
        dicts = dicts[0]
    factory = _get_factory(merge, kwargs)

    rv = factory()
    for d in dicts:

      rv.update(d)

E ValueError: dictionary update sequence element #0 has length 39; 2 is required ../../../miniconda/envs/test_env/lib/python3.6/site-packages/toolz/dicttoolz.py:39: ValueError ```

• [x] Closes #2512 (remove if there is no corresponding issue, which should only be the case for minor changes)
• [x] Tests added (for all bug fixes or enhancements)

• [x] Closes #1504, closes #1717, closes #1735
• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

1735 is stalled, so I jumped in.

I've added an error if cbar_ax is provided as an option for FacetGrid. Don't think it's really needed.

• [x] Closes #2381 (remove if there is no corresponding issue, which should only be the case for minor changes)
• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

Rotate x-axis dateticks by default, just as for plot.line()

• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Passes git diff upstream/master **/*py | flake8 --diff (remove if you did not edit any Python files)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

I like @fujiisoup's weighted mean demo in this stack overflow example: https://stackoverflow.com/questions/48510784/xarray-rolling-mean-with-weights

I thought it'd be a useful addition to the docs on rolling.