What happened?

From https://github.com/pydata/xarray/discussions/8951

if d is a MultiIndex-ed dataset with levels (x, y, z), and m is a dataset with a single coord x m.isel(x=d.x) builds a dataset with a MultiIndex with levels (y, z). This seems like it should work.

cc @benbovy

What did you expect to happen?

No response

Minimal Complete Verifiable Example

```Python import pandas as pd, xarray as xr, numpy as np

xr.set_options(use_flox=True)

test = pd.DataFrame() test["x"] = np.arange(100) % 10 test["y"] = np.arange(100) test["z"] = np.arange(100) test["v"] = np.arange(100)

d = xr.Dataset.from_dataframe(test) d = d.set_index(index = ["x", "y", "z"]) print(d)

m = d.groupby("x").mean() print(m)

print(d.xindexes) print(m.isel(x=d.x).xindexes)

xr.align(d, m.isel(x=d.x))

res = d.groupby("x") - m

print(res)

```

<xarray.Dataset> Dimensions: (index: 100) Coordinates: * index (index) object MultiIndex * x (index) int64 0 1 2 3 4 5 6 7 8 9 0 1 2 ... 8 9 0 1 2 3 4 5 6 7 8 9 * y (index) int64 0 1 2 3 4 5 6 7 8 9 ... 90 91 92 93 94 95 96 97 98 99 * z (index) int64 0 1 2 3 4 5 6 7 8 9 ... 90 91 92 93 94 95 96 97 98 99 Data variables: v (index) int64 0 1 2 3 4 5 6 7 8 9 ... 90 91 92 93 94 95 96 97 98 99 <xarray.Dataset> Dimensions: (x: 10) Coordinates: * x (x) int64 0 1 2 3 4 5 6 7 8 9 Data variables: v (x) float64 45.0 46.0 47.0 48.0 49.0 50.0 51.0 52.0 53.0 54.0 Indexes: ┌ index PandasMultiIndex │ x │ y └ z Indexes: ┌ index PandasMultiIndex │ y └ z ValueError...

MVCE confirmation

• [x] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [x] Complete example — the example is self-contained, including all data and the text of any traceback.
• [x] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [x] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [x] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

What happened?

Resetting a MultiIndex dim coordinate preserves the MultiIndex levels as IndexVariables. We should either reset the indexes for the multiindex level variables, or warn asking the users to do so

This seems to be the root cause exposed by https://github.com/pydata/xarray/pull/8809

cc @benbovy

What did you expect to happen?

No response

Minimal Complete Verifiable Example

```Python import numpy as np import xarray as xr

ND DataArray that gets stacked along a multiindex

da = xr.DataArray(np.ones((3, 3)), coords={"dim1": [1, 2, 3], "dim2": [4, 5, 6]}) da = da.stack(feature=["dim1", "dim2"])

Extract just the stacked coordinates for saving in a dataset

ds = xr.Dataset(data_vars={"feature": da.feature}) xr.testing.assertions._assert_internal_invariants(ds.reset_index(["feature", "dim1", "dim2"]), check_default_indexes=False) # succeeds xr.testing.assertions._assert_internal_invariants(ds.reset_index(["feature"]), check_default_indexes=False) # fails, but no warning either ```

What happened?

(xref #8658)

I'm not sure what the expected behaviour is here:

```python import xarray as xr import numpy as np from xarray.testing import _assert_internal_invariants

ds = xr.Dataset() ds.coords["1"] = ("1", np.array([1], dtype=np.uint32)) ds["1_"] = ("1", np.array([1], dtype=np.uint32)) ds = ds.rename_vars({"1": "0"}) ds ```

It looks like this sequence of operations creates a default index

But then ```python from xarray.testing import _assert_internal_invariants

_assert_internal_invariants(ds, check_default_indexes=True) fails with ... File ~/repos/xarray/xarray/testing/assertions.py:301, in _assert_indexes_invariants_checks(indexes, possible_coord_variables, dims, check_default) 299 if check_default: 300 defaults = default_indexes(possible_coord_variables, dims) --> 301 assert indexes.keys() == defaults.keys(), (set(indexes), set(defaults)) 302 assert all(v.equals(defaults[k]) for k, v in indexes.items()), ( 303 indexes, 304 defaults, 305 )

AssertionError: ({'0'}, set()) ```

What happened?

All tests are failing because of an error in create_test_data

from xarray.tests import create_test_data create_test_data()

```

AssertionError Traceback (most recent call last) Cell In[3], line 2 1 from xarray.tests import create_test_data ----> 2 create_test_data()

File ~/repos/xarray/xarray/tests/init.py:329, in create_test_data(seed, add_attrs, dim_sizes) 327 obj.coords["numbers"] = ("dim3", numbers_values) 328 obj.encoding = {"foo": "bar"} --> 329 assert all(var.values.flags.writeable for var in obj.variables.values()) 330 return obj

AssertionError: ```

What is your issue?

https://docs.xarray.dev/en/stable/getting-started-guide/installing.html#minimum-dependency-versions says that we follow NEP-29, and I think our min versions script also does that.

I propose we follow https://scientific-python.org/specs/spec-0000/

In practice, I think this means we mostly drop Python versions earlier.

What is your issue?

As we work through refactoring for NamedArray, it's pretty clear that Xarray will depend pretty closely on many files in namedarray/. For example various utils.py, pycompat.py, *ops.py, formatting.py, formatting_html.py at least. This promises to be quite painful if we did break NamedArray out in to its own repo (particularly around typing, e.g. https://github.com/pydata/xarray/pull/8309)

I propose we use pydata/xarray as a monorepo that serves two packages: NamedArray and Xarray. - We can move as much as is needed to have NamedArray be independent of Xarray, but Xarray will depend quite closely on many utility functions in NamedArray. - We can release both at the same time similar to dask and distributed. - We can re-evaluate if and when NamedArray grows its own community.

I'm opening this issue for discussion really.

I stumbled on autoray (Github) by @jcmgray which provides an abstract interface to a number of array types.

What struck me was the very general lazy compute system. This opens up the possibility of lazy-but-not-dask computation.

Related: https://github.com/pydata/xarray/issues/2298 https://github.com/pydata/xarray/issues/1725 https://github.com/pydata/xarray/issues/5081

Is your feature request related to a problem?

See https://github.com/pydata/xarray/pull/6213/files#r795716713

We check whether the by variable for groupby is unique and monotonically increasing. But the fast path would also apply to unique and monotonically decreasing variables.

Describe the solution you'd like

Update the condition to is_monotonic_increasing or is_monotonic_decreasing and add a test.

Describe alternatives you've considered

No response

Additional context

No response

Is your feature request related to a problem?

Here are two datasets, in one a is a data_var, in the other a is a coordinate variable. The following fails ``` python import xarray as xr

ds1 = xr.Dataset({"a": ('x', [1, 2, 3])}) ds2 = ds1.set_coords("a") ds2.update(ds1) with 649 ambiguous_coords = coord_names.intersection(noncoord_names) 650 if ambiguous_coords: --> 651 raise MergeError( 652 "unable to determine if these variables should be " 653 f"coordinates or not in the merged result: {ambiguous_coords}" 654 ) 656 attrs = merge_attrs( 657 [var.attrs for var in coerced if isinstance(var, (Dataset, DataArray))], 658 combine_attrs, 659 ) 661 return _MergeResult(variables, coord_names, dims, out_indexes, attrs)

MergeError: unable to determine if these variables should be coordinates or not in the merged result: {'a'} ```

Describe the solution you'd like

I think we should replace this error with a warning and arbitrarily choose to either convert a to a coordinate variable or a data variable.

Describe alternatives you've considered

No response

Additional context

No response

What happened?

Seen in #4972

``` python import xarray as xr import numpy as np

array = np.array([0. +0.j, 0.+np.nan * 1j], dtype=np.complex64) var = xr.Variable("x", array) print(var.mean().data) print(array.mean()) ```

0j (nan+nanj)

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

What is your issue?

We have some stale issue but not that many.

Can we restrict the bot to only issues that are untagged, or tagged as "usage question" or are not assigned to a "project" instead? This might reduce a lot of the noise.

What happened?

``` python import numpy as np import xarray as xr

ds = xr.Dataset( {"foo": (("x", "y", "z"), np.ones((3, 4, 2)))}, {"x": ["a", "b", "c"], "y": [1, 2, 3, 4]}, ) expected = ds.sum("z") stacked = ds.stack(space=["x", "y"])

broadcasted, _ = xr.broadcast(stacked, stacked.space)

stacked.sum("z").unstack("space") # works broadcasted.sum("z").unstack("space") # error ```

```

ValueError Traceback (most recent call last) Input In [13], in <module> 10 broadcasted, _ = xr.broadcast(stacked, stacked.space) 11 stacked.sum("z").unstack("space") ---> 12 broadcasted.sum("z").unstack("space")

File ~/work/python/xarray/xarray/core/dataset.py:4332, in Dataset.unstack(self, dim, fill_value, sparse) 4330 non_multi_dims = set(dims) - set(stacked_indexes) 4331 if non_multi_dims: -> 4332 raise ValueError( 4333 "cannot unstack dimensions that do not " 4334 f"have exactly one multi-index: {tuple(non_multi_dims)}" 4335 ) 4337 result = self.copy(deep=False) 4339 # we want to avoid allocating an object-dtype ndarray for a MultiIndex, 4340 # so we can't just access self.variables[v].data for every variable. 4341 # We only check the non-index variables. 4342 # https://github.com/pydata/xarray/issues/5902

ValueError: cannot unstack dimensions that do not have exactly one multi-index: ('space',) ```

What did you expect to happen?

This should work.

Minimal Complete Verifiable Example

No response

Relevant log output

No response

Anything else we need to know?

No response

Environment

xarray main after the flexible indexes refactor

What happened?

``` python import xarray as xr

xr.where(xr.DataArray([1, 2, 3]) > 0, 1, 0) ```

fails with

`` 1809 if keep_attrs is True: 1810 # keep the attributes of x, the second parameter, by default to 1811 # be consistent with thewheremethod ofDataArrayandDataset` -> 1812 keep_attrs = lambda attrs, context: attrs[1] 1814 # alignment for three arguments is complicated, so don't support it yet 1815 return apply_ufunc( 1816 duck_array_ops.where, 1817 cond, (...) 1823 keep_attrs=keep_attrs, 1824 )

IndexError: list index out of range ```

The workaround is to pass keep_attrs=False

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

Relevant log output

No response

Anything else we need to know?

No response

Environment

xarray 2022.3.0

version rasterio 1.1.1 py36h900e953_0 conda-forge

``` =================================== FAILURES =================================== ___ TestRasterio.testrasterio_vrt ____

self = <xarray.tests.test_backends.TestRasterio object at 0x7fc8355c8f60>

def test_rasterio_vrt(self):
    import rasterio

    # tmp_file default crs is UTM: CRS({'init': 'epsg:32618'}
    with create_tmp_geotiff() as (tmp_file, expected):
        with rasterio.open(tmp_file) as src:
            with rasterio.vrt.WarpedVRT(src, crs="epsg:4326") as vrt:
                expected_shape = (vrt.width, vrt.height)
                expected_crs = vrt.crs
                expected_res = vrt.res
                # Value of single pixel in center of image
                lon, lat = vrt.xy(vrt.width // 2, vrt.height // 2)

              expected_val = next(vrt.sample([(lon, lat)]))

xarray/tests/test_backends.py:3966:

/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/rasterio/sample.py:43: in sample_gen data = read(indexes, window=window, masked=masked, boundless=True)

??? E ValueError: WarpedVRT does not permit boundless reads

rasterio/_warp.pyx:978: ValueError ```

By switching to numpy_groupies we are vectorizing our groupby reductions. I think we can do the same for groupby's binary ops.

Here's an example array ``` python import numpy as np import xarray as xr

%load_ext memory_profiler

N = 4 * 2000 da = xr.DataArray( np.random.random((N, N)), dims=("x", "y"), coords={"labels": ("x", np.repeat(["a", "b", "c", "d", "e", "f", "g", "h"], repeats=N//8))}, ) ```

Consider this "anomaly" calculation, anomaly defined relative to the group mean

``` python def anom_current(da): grouped = da.groupby("labels") mean = grouped.mean() anom = grouped - mean return anom

```

With this approach, we loop over each group and apply the binary operation: https://github.com/pydata/xarray/blob/a1635d324753588e353e4e747f6058936fa8cf1e/xarray/core/computation.py#L502-L525

This saves some memory, but becomes slow for large number of groups.

We could instead do def anom_vectorized(da): mean = da.groupby("labels").mean() mean_expanded = mean.sel(labels=da.labels) anom = da - mean_expanded return anom

Now we are faster, but construct an extra array as big as the original array (I think this is an OK tradeoff). ``` %timeit anom_current(da)

1.4 s ± 20.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit anom_vectorized(da)

937 ms ± 5.26 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```

(I haven't experimented with dask yet, so the following is just a theory).

I think the real benefit comes with dask. Depending on where the groups are located relative to chunking, we could end up creating a lot of tiny chunks by splitting up existing chunks. With the vectorized approach we can do better.

Ideally we would reindex the "mean" dask array with a numpy-array-of-repeated-ints such that the chunking of mean_expanded exactly matches the chunking of da along the grouped dimension.

~In practice, dask.array.take doesn't allow specifying "output chunks" so we'd end up chunking "mean_expanded" based on dask's automatic heuristics, and then rechunking again for the binary operation.~

Thoughts?

cc @rabernat

What happened?

After vectorized indexing a DataArray with dim xby a DataArray z, we get a DataArray with dim z and x as non-dim coordinate. But x is still an IndexVariable, not a normal variable.

What did you expect to happen?

x should be a normal variable.

Minimal Complete Verifiable Example

```python import xarray as xr xr.set_options(display_style="text")

da = xr.DataArray([1, 2, 3], dims="x", coords={"x": [0, 1, 2]}) idxr = xr.DataArray([1], dims="z", name="x", coords={"z": ("z", ["a"])}) da.sel(x=idxr) ```

<xarray.DataArray (z: 1)> array([2]) Coordinates: x (z) int64 1 * z (z) <U1 'a'

x is a non-dim coordinate but is backed by a IndexVariable with the wrong name! python da.sel(x=idxr).x.variable

<xarray.IndexVariable 'z' (z: 1)> array([1])

Relevant log output

No response

Anything else we need to know?

No response

Environment

xarray main but this bug was present prior to the explicit indexes refactor.

see https://github.com/pydata/xarray/pull/3285

Continuing discussion form #3491.

Pandas has added attrs to their objects. We should decide on what to do with them in the DataArray constructor. Many tests fail if we don't handle this case explicitly.

@dcherian:

Not sure what we want to do about these attributes in the long term. One option would be to pop the name attribute, assign to DataArray.name and keep the rest as DataArray.attrs? But what if name clashes with the provided name?

@max-sixty:

Agree! I think we could prioritize the supplied name above that in attrs. Another option would be raising an error if both were supplied.

The RTD build is failing on all PRs with

Sphinx parallel build error: nbsphinx.NotebookError: UndefinedError in examples/ERA5-GRIB-example.ipynb: 'nbformat.notebooknode.NotebookNode object' has no attribute 'tags'

What happened: The formatting of section headings "for users", "community" etc. has changed: https://xarray.pydata.org/en/latest/

Just as a reminder.

Right now all rolling operations with dask arrays use .construct().reduce().

Downstream issue: https://github.com/xarray-contrib/cf-xarray/issues/57 Example: no reprs displayed in https://cf-xarray.readthedocs.io/en/latest/examples/introduction.html

@benbovy's diagnosis:

It looks like bootstrap 4 (used by sphinx-book-theme) forces all html elements with hidden attributes to be actually hidden (source), so the hack in pydata/xarray#4053 does not work here (the result is even worse). I guess that a workaround would be to add some custom CSS such as .xr-wrap { display: block !important }, assuming that custom CSS is loaded after Bootstrap's CSS. Not ideal, though, it looks like a hack on top of another hack.

Is your feature request related to a problem? Please describe. I have datasets with lots of non-dim coord variables. Its annoying to search through and look at the dimension coordinates to get an idea of what subset of data I am looking at.

Describe the solution you'd like I think we should show dimension coordinate variables at the top of the coordinates repr.

Example code

python ds = xr.Dataset() ds.coords["as"] = 10 ds["var"] = xr.DataArray(np.ones((10,)), dims="x", coords={"x": np.arange(10)}) ds

Related #4409

See https://readthedocs.org/projects/xray/builds/

There's no useful information in the logs AFAICT: e.g. https://readthedocs.org/projects/xray/builds/11504571/

flake8 is failing on master (https://dev.azure.com/xarray/xarray/_build/results?buildId=2820&view=logs&jobId=a577607c-d99b-546f-eeb4-2341e9a21630&j=a577607c-d99b-546f-eeb4-2341e9a21630&t=7308a173-bf34-5af1-b6d9-30c4d79bebeb) with

``` ========================== Starting Command Output =========================== /bin/bash --noprofile --norc /home/vsts/work/_temp/e6322963-dd1c-4887-ba6a-2aa7ec888f4c.sh ./xarray/backends/memory.py:43:32: E741 ambiguous variable name 'l' ./xarray/backends/common.py:244:32: E741 ambiguous variable name 'l'

[error]Bash exited with code '1'.

Finishing: flake8 lint checks ```

Looks like the last run was on Thursday: https://dev.azure.com/xarray/xarray/_build?definitionId=1&_a=summary&view=runs

No tests have been run for PRs #3826 #3836 #3858 and #3807 despite these having been updated recently.

There is a workaround posted at this Azure issue: https://status.dev.azure.com/_event/179641421 but it looks like a fix is coming soon.

It would be good to link to https://docs.dask.org/en/latest/array-chunks.html in https://xarray.pydata.org/en/stable/dask.html#chunking-and-performance

Sample error from https://travis-ci.org/pydata/xarray/jobs/466431752

``` _____ test_dataarray_with_dask_coords ______ def test_dataarray_with_dask_coords(): import toolz x = xr.Variable('x', da.arange(8, chunks=(4,))) y = xr.Variable('y', da.arange(8, chunks=(4,)) * 2) data = da.random.random((8, 8), chunks=(4, 4)) + 1 array = xr.DataArray(data, dims=['x', 'y']) array.coords['xx'] = x array.coords['yy'] = y

    assert dict(array.__dask_graph__()) == toolz.merge(data.__dask_graph__(),
                                                       x.__dask_graph__(),
                                                       y.__dask_graph__())

  (array2,) = dask.compute(array)

xarray/tests/test_dask.py:824:

../../../miniconda/envs/test_env/lib/python3.6/site-packages/dask/base.py:395: in compute dsk = collections_to_dsk(collections, optimize_graph, *kwargs) ../../../miniconda/envs/test_env/lib/python3.6/site-packages/dask/base.py:187: in collections_to_dsk for opt, val in groups.items()} ../../../miniconda/envs/test_env/lib/python3.6/site-packages/dask/base.py:187: in <dictcomp> for opt, val in groups.items()} ../../../miniconda/envs/test_env/lib/python3.6/site-packages/dask/base.py:212: in _extract_graph_and_keys graph = merge(graphs)

dicts = <dask.sharedict.ShareDict object at 0x7f307d29a128>, kwargs = {} factory = <class 'dict'>, rv = {} d = ('arange-36f53ab1e6153a63bbf7f4f8ff56693c', 0) def merge(dicts, *kwargs): """ Merge a collection of dictionaries

    >>> merge({1: 'one'}, {2: 'two'})
    {1: 'one', 2: 'two'}

    Later dictionaries have precedence

    >>> merge({1: 2, 3: 4}, {3: 3, 4: 4})
    {1: 2, 3: 3, 4: 4}

    See Also:
        merge_with
    """
    if len(dicts) == 1 and not isinstance(dicts[0], dict):
        dicts = dicts[0]
    factory = _get_factory(merge, kwargs)

    rv = factory()
    for d in dicts:

      rv.update(d)

E ValueError: dictionary update sequence element #0 has length 39; 2 is required ../../../miniconda/envs/test_env/lib/python3.6/site-packages/toolz/dicttoolz.py:39: ValueError ```