• [x] Closes #8628
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

In addition to the bi-weekly community developers meeting, we plan to have 30min meetings on a weekly basis -- every Tue 8:30-9:00 PDT (17:30-18:00 CEST) -- to discuss the flexible indexes refactor.

Anyone from @pydata/xarray feel free to join! The first meeting is in a couple of hours.

Zoom link (subject to change).

Google calendar

Meeting notes

Is it safe to use xarray.core.variable.as_variable() externally? I guess that currently it is not.

I have a specific use case where this would be very useful.

I'm working on a package that heavily uses and extends xarray for landscape evolution modeling, and inside a custom class for model parameters I want to be able to create xarray.Variable objects on the fly from any provided object, e.g., a scalar value, an array-like, a (dims, data[, attrs]) tuple, another xarray.Variable, a xarray.DataArray... exactly what xarray.core.variable.as_variable() does.

Although I know that Variable objects are not needed in most use cases, in this specific case a clean solution would be the following

```python import xarray as xr

class Parameter(object):

def to_variable(self, obj):
    return xr.as_variable(obj)
    # ... some validation logic on, e.g., data type, value bounds, dimensions...
    # ... add default attributes to the created variable (e.g., units, description...)

```

I don't think it is a viable option to copy as_variable() and all its dependent code in my package as it seems to have quite a lot of logic implemented.

A workaround using only public API would be something like:

```python class Parameter(object):

def to_variable(self, obj):
    return xr.Dataset(data_vars={'v': obj}).variables['v']

```

but it feels a bit hacky.

• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

After working more on Coordinates I realize that the default behavior of its constructor could be more consistent with other Xarray objects. This PR changes this default behavior such that:

• Pandas indexes are created for dimension coordinates if indexes=None (default). To create dimension coordinates with no index, just pass indexes={}.
• If another Coordinates object is passed as input, its indexes are also added to the new created object. Since we don't support alignment / merge here, the following call raises an error: xr.Coordinates(coords=xr.Coordinates(...), indexes={...}).

This PR introduces a breaking change since Coordinates are now exposed in v2023.8.0, which has just been released. It is a bit unfortunate but I think it may be OK for a fresh feature, especially if the next release will be soon after this one.

I wanted to fix the following error with mypy 1.5:

xarray/core/dataset.py:505: error: Definition of "__eq__" in base class "DatasetOpsMixin" is incompatible with definition in base class "Mapping" [misc]

Which looks similar to https://github.com/python/mypy/issues/9319. It is weird that here it worked with mypy versions < 1.5, though.

I don't know if there is a better fix, but I thought that redefining __eq__ in Dataset would be a bit less dirty workaround than adding type: ignore in the class declaration.

• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

This is consistent with the Dataset and DataArray assign methods (now that Coordinates is also exposed as public API).

This allows writing:

```python midx = pd.MultiIndex.from_arrays([["a", "a", "b", "b"], [0, 1, 0, 1]]) midx_coords = xr.Coordinates.from_pandas_multiindex(midx, "x")

ds = xr.Dataset(coords=midx_coords.assign(y=[1, 2])) ```

which is quite common (at least in the tests) and a bit nicer than

python ds = xr.Dataset(coords=midx_coords.merge({"y": [1, 2]}).coords)

Another item in #719.

I added tests and updated the docs, so this is ready for review.

• [x] Closes #4091
• [x] Tests added

The message suggests using .expand_dims().

• [x] Closes many issues:
• [x] closes #1366
• [x] closes #1408
• [x] closes #2489
• [x] closes #3432
• [x] closes #4542
• [x] closes #4955
• [x] closes #5202
• [x] closes #5645
• [x] closes #5691
• [x] closes #5697
• [x] closes #5700
• [x] closes #5727
• [x] closes #5953
• [x] closes #6183
• [x] closes #6313
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst (new Index and Indexes API not public yet)

Follow-up on #5636 (work in progress), supersedes #2195.

This is likely to be going big, sorry in advance! It'll be safer to make a release before merging this PR.

Current progress:

• [x] create (default) indexes using the Index classes
• [x] refactor default indexes created when 1st accessing .xindexes or .indexes
• [x] support for non-default indexes (no public API yet)
• [x] remove multi-index virtual coordinates (replace it by regular coordinates)
• [x] refactor internal (text / html) formatting functions
• [x] internal refactor of location-based selection (.isel())
• [x] internal refactor of label-based selection (.sel())
• [x] internal refactor of .rename()
• Some changes in behavior (see comments below)
  • see #4108
  • see #4107
  • see #4417
• [x] internal refactor of set_index / reset_index
• [x] internal refactor of stack / unstack
  • Some changes in behavior (see comments below)
• [x] internal refactor of Dataset.to_stacked_array
• [x] internal refactor of swap_dims
• [x] internal refactor of expand_dims
• [x] internal refactor of alignment
• [x] internal refactor of reindex and reindex_like
• [x] internal refactor of interp and interp_like
• [x] internal refactor of merge
• [x] internal refactor of concat
• [x] internal refactor of computation
• [x] internal refactor of copy
• [x] internal refactor of update, assign, __setitem__, del, drop_vars, etc.
  • updates must not corrupt multi-coordinate indexes
• [x] internal refactor of set_coords and reset_coords
• internal refactor of drop_sel and drop_isel (maybe later)
• [x] internal refactor of pad
• [x] internal refactor of shift
• [x] internal refactor of roll

TODO:

• [x] Uniformize Index API with Xarray's API
  • [x] rename Index.query() -> Index.sel()?
  • [x] rename PandasMultiIndex.from_product() -> PandasMultiIndex.stack()? Add Index.stack() and Index.unstack().
  • [x] remove Index.union() and Index.intersection()
• [x] Use Index.create_variables() internally
  • [x] remove PandasIndex.from_pandas_index() and PandasMultiIndex.from_pandas_index() (use constructor + .create_variables() instead)
• [x] Review where .xindexes is used and use private API instead (._indexes) if possible for speed
  • [x] requires that _indexes always returns a mapping
• [x] Use from __future__ import annotations in indexes.py
• [x] Re-activate default indexes invariant check (with opt-out for some tests)

In next PRs:

• custom Index.__repr__ and Index._repr_inline_
• add an Indexes section in DataArray / Dataset reprs
• update public API (set_index, reset_index, drop_indexes, Dataset and DataArray constructors, etc.)
• allow multi-dimensional variables with name in var.dims

• [x] Closes #5553
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

This implements option 3 (sort of) described in https://github.com/pydata/xarray/issues/5553#issue-933551030:

• the goal is to avoid wrapping an xarray.Index into an xarray.Variable and keep those two concepts distinct from each other.
• the xarray.Index.from_variables class constructor accepts a dictionary of xarray.Variable objects as argument and may (or should?) also return corresponding xarray.IndexVariable objects to ensure immutability.
• for PandasIndex, the new returned xarray.IndexVariable wraps the underlying pd.Index via a PandasIndexingAdapter (this reverts some changes made in #5102).
• for PandasMultiIndex, this PR adds PandasMultiIndexingAdapter so that we can wrap the pandas multi-index in separate coordinate variables objects: one for the dimension + one for each level. The level coordinates data internally hold a reference to the dimension coordinate data to avoid indexing the same underlying pd.MultiIndex for each of those coordinates (PandasMultiIndexingAdapter.__getitem__ is memoized for that purpose).

This is very much work in progress, I need to update (or revert) all related parts of Xarray's internals, update tests, etc. At this stage any comment on the approach described above is welcome.

• [x] Closes #7214
• [x] Closes #6392
• [x] xref #6633
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

This is a rework of #7214. It follows the suggestions made in https://github.com/pydata/xarray/pull/7214#issuecomment-1295283938, https://github.com/pydata/xarray/pull/7214#issuecomment-1297046405 and https://github.com/pydata/xarray/pull/7214#issuecomment-1293774799:

• No indexes argument is added to Dataset.__init__, and the indexes argument of DataArray.__init__ is kept private (i.e., valid only if fastpath=True)
• When a Coordinates object is passed to a new Dataset or DataArray via the coords argument, both coordinate variables and indexes are copied/extracted and added to the new object
• This PR also adds ~~an IndexedCoordinates subclass~~ Coordinates public constructors used to create Xarray coordinates and indexes from non-Xarray objects. For example, the Coordinates.from_pandas_multiindex() class method creates a new set of index and coordinates from an existing pd.MultiIndex.

EDIT: IndexCoordinates has been merged with Coordinates

EDIT2: it ended up as a pretty big refactor with the promotion of Coordinates has a 2nd-class Xarray container that supports alignment like Dataset and DataArray. It is still quite advanced API, useful for passing coordinate variables and indexes around. Internally, Coordinates objects are still "virtual" containers (i.e., proxies for coordinate variables and indexes stored in their corresponding DataArray or Dataset objects). For now, a "stand-alone" Coordinates object created from scratch wraps a Dataset with no data variables.

Some examples of usage:

```python import pandas as pd import xarray as xr

midx = pd.MultiIndex.from_product([["a", "b"], [1, 2]], names=("one", "two"))

coords = xr.Coordinates.from_pandas_multiindex(midx, "x")

Coordinates:

* x (x) object MultiIndex

* one (x) object 'a' 'a' 'b' 'b'

* two (x) int64 1 2 1 2

ds = xr.Dataset(coords=coords)

<xarray.Dataset>

Dimensions: (x: 4)

Coordinates:

* x (x) object MultiIndex

* one (x) object 'a' 'a' 'b' 'b'

* two (x) int64 1 2 1 2

Data variables:

empty

ds_to_be_deprecated = xr.Dataset(coords={"x": midx}) ds_to_be_deprecated.identical(ds)

True

da = xr.DataArray([1, 2, 3, 4], dims="x", coords=ds.coords)

<xarray.DataArray (x: 4)>

array([1, 2, 3, 4])

Coordinates:

* x (x) object MultiIndex

* one (x) object 'a' 'a' 'b' 'b'

* two (x) int64 1 2 1 2

```

TODO:

• [x] update assign_coords too so it has the same behavior if a Coordinates object is passed?
• [x] How to avoid building any default index? It seems silly to add or use the indexes argument just for that purpose? ~~We could address that later.~~ Solution: wrap the coordinates dict in a Coordinates objects, e.g., ds = xr.Dataset(coords=xr.Coordinates(coords_dict)).

@shoyer, @dcherian, anyone -- what do you think about the approach proposed here? I'd like to check that with you before going further with tests, docs, etc.

• [x] Closes #6392
• [x] Closes #6633 ?
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

From https://github.com/pydata/xarray/issues/6392#issuecomment-1290454937:

I'm thinking of only accepting one or more instances of Indexes as indexes argument in the Dataset and DataArray constructors. The only exception is when fastpath=True a mapping can be given directly. Also, when an empty collection of indexes is passed this skips the creation of default pandas indexes for dimension coordinates.

• It is much easier to handle: just check that keys returned by Indexes.variables do no conflict with the coordinate names in the coords argument
• It is slightly safer: it requires the user to explicitly create an Indexes object, thus with less chance to accidentally provide coordinate variables and index objects that do not relate to each other (we could probably add some safe guards in the Indexes class itself)
• It is more convenient: an Xarray Index may provide a factory method that returns an instance of Indexes that we just need to pass as indexes, and we could also do something like ds = xr.Dataset(indexes=other_ds.xindexes)

• [x] Closes #7405
• [x] Closes #7588
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

This PR documents the API of the Index base class and adds a guide for creating custom indexes (reworked from https://hackmd.io/Zxw_zCa7Rbynx_iJu6Y3LA). Hopefully it will help anyone experimenting with this feature.

@pydata/xarray your feedback would be very much appreciated! I've been into this for quite some time, so there may be things that seem obvious to me but that you can still find very confusing or non-intuitive. It would then deserve some extra or better explanation.

More specifically, I'm open to any suggestion on how to better illustrate this with clear and succinct examples.

There are other parts of the documentation that still need to be updated regarding the indexes refactor (e.g., "dimension" coordinates, xindexes property, set/drop indexes, etc.). But I suggest to do that in separate PRs and focus here on creating custom indexes.

• [x] Closes #7563
• [x] Closes #8039
• [x] Closes #8056
• [x] Closes #7885
• [x] Closes #7921
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

This refactor should better handle multi-coordinate indexes when updating (or assigning) new coordinates.

It also fixes, better isolates and better warns a bunch of deprecated pandas multi-index special cases (i.e., directly passing pd.MultiIndex objects or updating a multi-index dimension coordinate). I very much look forward to seeing support for those cases dropped :).

• [x] Benchmark added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

May fix some performance regressions, e.g., see https://github.com/pydata/xarray/issues/7376#issuecomment-1352989233.

@ravwojdyla with this PR ds.assign(foo=~ds["d3"]) in your example should be much faster (on par with version 2022.3.0).

• [x] Closes
• [x] fixes #6946
• [x] fixes #6989
• [x] fixes #6959
• [x] fixes #6969
• [x] fixes #7036
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Restore behavior prior to the explicit indexes refactor (i.e., refactored but without breaking changes).

TODO:

• [x] review set_index
• [x] review reset_index

For reset_index, the only behavior that is not restored here is the coordinate renamed with a _ suffix when dropping a single index. This was originally to prevent any coordinate with no index matching a dimension name, which is now irrelevant. That is a quite dirty workaround and I don't know who is relying on it (no complaints yet), but I'm open to restore it if needed (esp. considering that we may later deprecate reset_index completely in favor of drop_indexes #6971).

Another thing that I've noticed while working on #5692.

Currently it is not possible to have a Dataset with a same name used for both a dimension and a multi-index level. I guess the reason is to prevent some errors like unmatched dimension sizes when eventually the multi-index is dropped with renamed dimension(s) according to the level names (e.g., with sel or unstack). See #2299.

I'm wondering how we should handle this in the context of flexible / custom indexes:

A. Keep this current behavior as a special case for (pandas) multi-indexes. This would avoid breaking changes but how to support custom indexes that could eventually be used like pandas multi-indexes in sel or stack?

B. Introduce some tag in xarray.Index so that we can identify a multi-coordinate index that behaves like a hierarchical index (i.e., levels may be dropped into a single index/coordinate with dimension renaming)

C. Do not allow any dimension name matching the name of a coordinate attached to a multi-coordinate index. This seems silly?

D. Eventually revert #2353 and let users taking care of potential conflicts.

Is your feature request related to a problem?

This is part of #6293 (explicit indexes next steps).

Describe the solution you'd like

A Mapping[Hashable, Index] would probably be the most obvious (optional) value type accepted for the indexes argument of the Dataset and DataArray constructors.

pros:

• consistent with the xindexes property

cons:

• need to be careful with what is passed as coords and indexes
• multi-indexes: redundancy and order matters (e.g., pandas multi-index levels)

An example with a pandas multi-index

Currently a pandas multi-index may be passed directly as one (dimension) coordinate ; it is then "unpacked" into one dimension (tuple values) coordinate and one or more level coordinates. I would suggest depreciating this behavior in favor of a more explicit (although more verbose) way to pass an existing pandas multi-index:

```python import pandas as pd import xarray as xr

pd_idx = pd.MultiIndex.from_product([["a", "b"], [1, 2]], names=("foo", "bar")) idx = xr.PandasMultiIndex(pd_idx, "x")

indexes = {"x": idx, "foo": idx, "bar": idx} coords = idx.create_variables()

ds = xr.Dataset(coords=coords, indexes=indexes) ```

The cases below should raise an error:

```python ds = xr.Dataset(indexes=indexes)

ValueError: missing coordinate(s) for index(es): 'x', 'foo', 'bar'

ds = xr.Dataset( coords=coords, indexes={"x": idx, "foo": idx}, )

ValueError: missing index(es) for coordinate(s): 'bar'

ds = xr.Dataset( coords={"x": coords["x"], "foo": [0, 1, 2, 3], "bar": coords["bar"]}, indexes=indexes, )

ValueError: conflict between coordinate(s) and index(es): 'foo'

ds = xr.Dataset( coords=coords, indexes={"x": idx, "foo": idx, "bar": xr.PandasIndex([0, 1, 2], "y")}, )

ValueError: conflict between coordinate(s) and index(es): 'bar'

```

Should we raise an error or simply ignore the index in the case below?

```python ds = xr.Dataset(coords=coords)

ValueError: missing index(es) for coordinate(s): 'x', 'foo', 'bar'

or

create unindexed coordinates 'foo' and 'bar' and a 'x' coordinate with a single pandas index

```

Should we silently reorder the coordinates and/or indexes when the levels are not passed in the right order? It seems odd requiring mapping elements be passed in a given order.

```python ds = xr.Dataset(coords=coords, indexes={"bar": idx, "x": idx, "foo": idx}) list(ds.xindexes.keys())

["x", "foo", "bar"]

```

How to generalize to any (custom) index?

With the case of multi-index, it is pretty easy to check whether the coordinates and indexes are consistent because we ensure consistent pd_idx.names vs. coordinate names and because idx.get_variables() returns Xarray IndexVariable objects where variable data wraps the pandas multi-index.

However, this may not be easy for other indexes. Some Xarray custom indexes (like a KD-Tree index) likely won't return anything from .get_variables() as they don't support wrapping internal data as coordinate data. Right now there's nothing in the Xarray Index base class that could help checking consistency between indexes vs. coordinates for any kind of index.

How could we solve this?

• A. add a .coords property to the Xarray Index base class, that returns a dict[Hashable, IndexVariable].

  • Ambiguous when an Index is created directly, i.e., like above xr.PandasMultiIndex(pd_idx, "x"). Should .coords return None and return the coordinates returned by the last .get_variables() call?
  • What if different sets of coordinates refer to a common index (e.g., after copying the coordinate variables, etc.)?

• B. add a .coord_names property to the Xarray Index base class that returns tuple[Hashable, ...], and add a private attribute to IndexVariable that returns the index object (or return it via a very lightweight IndexAdapter base class used to wrap variable data).

  • Index.get_variables(variables) would by default return shallow copies of the input variables with a reference to the index object.
  • If that's necessary, we could also store the coordinate dimensions in coord_names, i.e., using tuple[tuple[Hashable, tuple[Hashable, ...]], ...].

I think I prefer the second option.

Describe alternatives you've considered

Also allow passing index types (and build options) via indexes

I.e., Mapping[Hashable, Index | Type[Index] | tuple[TypeIndex, Mapping[Any, Any]]], so that new indexes can be created from the passed coordinates at DataArray or Dataset creation.

pros:

• Flexible.

cons:

• This is complicated. Constructing the Dataset / DataArray (with default indexes) first then calling .set_index is probably better.
• Hard to deal with multi-index (redundancy of build option, etc.)

Pass multi-indexes once, grouped by coordinate names

I.e., indexes keys accept tuples: Mapping[Hashable | tuple[Hashable, ...], Index]

pros:

• No redundancy and easier to check consistency between indexes vs. coordinates

cons:

• Not consistent with the .xindexes property
• Complicated when eventually using tuples for coordinate names?

Additional context

No response

It allows passing options to the constructor of a custom Index subclass, in case there's any relevant build options to expose to users. This could for example be the distance metric chosen for an index based on sklearn.neighbors.BallTree, or the CRS definition for a geospatial index.

The **options arguments of Dataset.set_xindex() are passed through.

An alternative way would be to pass options via coordinate metadata, like the spatial_ref coordinate in rioxarray. Perhaps both alternatives may co-exist?

This PR also adds type annotations to set_xindex().

• [x] Closes #6849
• [x] Supersedes #6800
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

This PR adds Dataset and DataArray .set_xindex and .drop_indexes methods (the latter is also discussed in #4366). I've cherry picked the relevant commits in the scipy22 branch and added a few more commits. This PR also allows passing build options to any Index.

Some comments and open questions:

• Should we make the index_cls argument of set_xindex optional?
• I.e., set_index(coord_names, index_cls=None, **options) where a pandas index is created by default (or a pandas multi-index if several coordinate names are given), provided that the coordinate(s) are valid 1-d candidates.

• This would be redundant with the existing set_index method, but this would be convenient if we later depreciate it.

• Should we depreciate set_index and reset_index? I think we should, but probably not at this point yet.

• There's a special case for multi-indexes where set_xindex(["foo", "bar"], PandasMultiIndex) adds a dimension coordinate in addition to the "foo" and "bar" level coordinates so that it is consistent with the rest of Xarray. I find it a bit annoying, though. Probably another motivation for depreciating this dimension coordinate.

• In this PR I also imported the Index base class in Xarray's root namespace.

• It is needed for custom indexes and it's just a little more convenient than importing it from xarray.core.indexes.
• Should we do the same for PandasIndex and PandasMultiIndex subclasses? Maybe if one wants to create a custom index inheriting from it. PandasMultiIndex factory methods could be also useful if we depreciate passing pd.MultiIndex objects as DataArray / Dataset coordinates.

• [x] Closes #6607
• [x] Closes #4107
• [x] Closes #6229
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Current implemented "fix": raise a UserWarning and suggest using swap_dims (*)

Alternatively, we could:

• revert the breaking change (i.e., create the index again) and raise a DeprecationWarning instead
• raise an error instead of a warning

I don't have strong opinions on this, I'm happy to implement another alternative. The downside of reverting the breaking change now is that unfortunately it will introduce a breaking change in the next release., while workarounds are pretty straightforward.

(*) from https://github.com/pydata/xarray/issues/6607#issuecomment-1126587818, doing ds.set_coords(['lon']).rename(x='lon').set_index(lon='lon') is working too. With #6971, .set_xindex('lon') could work as well.

• [x] Closes #6987
• [x] Tests added

• [x] Closes #6836
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] Closes #6434
• [x] Tests added

• [x] Closes #7097
• [x] Tests added

@dcherian the DeprecationWarning was ignored by default for .assign_coords() because of https://github.com/pydata/xarray/pull/6798#discussion_r924653224. I changed it to FutureWarning so that it is shown for both .assign() and .assign_coords().

• [x] Closes #6852
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] Closes #7346
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

What happened?

See https://github.com/martinfleis/xvec/issues/13#issue-1472023524

What did you expect to happen?

assign_coords() should preserve the index of coordinates that are not updated or not part of a dropped multi-coordinate index.

Minimal Complete Verifiable Example

See https://github.com/martinfleis/xvec/issues/13#issue-1472023524

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

What is your issue?

xref https://github.com/pydata/xarray/pull/6795#discussion_r932665544 and #6293 (Public API section).

The scipy22 branch contains the addition of a .set_xindex() method to DataArray and Dataset so that participants at the SciPy 2022 Xarray sprint could experiment with custom indexes. After thinking more about it, I'm wondering if it couldn't actually be part of Xarray's public API alongside .set_index() (at least for a while).

• Having two methods .set_xindex() vs. .set_index() would be quite consistent with the .xindexes vs. .indexes properties that are already there.

• I actually like the .set_xindex() API proposed in the scipy22, i.e., setting one index at a time from one or more coordinates, possibly with build options. While it could be possible to support both that and .set_index()'s current API (quite specific to pandas multi-indexes) all in one method, it would certainly result in a much more confusing API and internal implementation.

• In the long term we could progressively get rid of .indexes and .set_index() and/or rename .xindexes to .indexes and .set_xindex() to .set_index().

Thoughts @pydata/xarray?

What happened?

Resetting a multi-index to a single level (i.e., a single index) does not rename the remaining level coordinate to the dimension name.

What did you expect to happen?

While it is certainly more consistent not to rename the level coordinate here (since an index can be assigned to a non-dimension coordinate now), it breaks from the old behavior. I think it's better not introduce any breaking change. As discussed elsewhere, we might eventually want to deprecate reset_index in favor of drop_indexes (#6971).

Minimal Complete Verifiable Example

```Python import pandas as pd import xarray as xr

midx = pd.MultiIndex.from_product([["a", "b"], [1, 2]], names=("foo", "bar"))

ds = xr.Dataset(coords={"x": midx})

<xarray.Dataset>

Dimensions: (x: 4)

Coordinates:

* x (x) object MultiIndex

* foo (x) object 'a' 'a' 'b' 'b'

* bar (x) int64 1 2 1 2

Data variables:

empty

rds = ds.reset_index("foo")

v2022.03.0

<xarray.Dataset>

Dimensions: (x: 4)

Coordinates:

* x (x) int64 1 2 1 2

foo (x) object 'a' 'a' 'b' 'b'

Data variables:

empty

v2022.06.0

<xarray.Dataset>

Dimensions: (x: 4)

Coordinates:

foo (x) object 'a' 'a' 'b' 'b'

* bar (x) int64 1 2 1 2

Dimensions without coordinates: x

Data variables:

empty

```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

@benbovy I also just tested the get_unique() method that you mentioned and maybe noticed a related issue here, which I'm not sure is wanted / expected.

Taking the above dataset ds, accessing this function results in an error:

```python

ds.indexes.get_unique()

TypeError: unhashable type: 'MultiIndex' ```

However, for xindexes it works: ```python

ds.xindexes.get_unique()

[<xarray.core.indexes.PandasMultiIndex at 0x7f105bf1df20>] ```

Originally posted by @lukasbindreiter in https://github.com/pydata/xarray/issues/6752#issuecomment-1236717180

Recent and ongoing developments in xarray turn DataArray and Dataset more and more into data wrappers that are extensible at (almost) every level:

• domain-specific methods (accessors)
• io backends (netcdf, raster, zarr, etc.) via an abstract DataStore interface
• array backends (numpy, dask, sparse) via multidispatch or hooks (#1938)
• soon custom indexes? (kd-tree, out-of-core indexes... #1603, #1650, #475)

Regarding the latter, I’m thinking about the idea of extending xarray at an even more abstract level, i.e., the possibility of adding / registering "coordinate wrappers" to DataArray or Dataset objects. Basically, it would correspond to adding any object that allows to do some operation based on one or several coordinates ~~(I haven’t found any better name than "coordinate agent" to describe that)~~.

EDIT: "coordinate agents" may not be quite right here, I changed that to "coordinate wrappers")

Indexes are a specific case of coordinate wrappers that serve the purpose of indexing. This is built in xarray.

While indexing is enough in 80% of cases, I see a couple of use cases where other coordinate wrappers (built outside of xarray) would be nice to have:

• Grids. For example, xgcm implements operations (interp, diff) on physical axes that may each include several coordinates, depending on the position of the coordinate labels on the axis (center, left…). Other grids define their topology using a greater number of coordinates (e.g., ugrid). Storing regridding weights might be another use case?
• Clocks. For example, xarray-simlab use one or several coordinates to define the timeline of a computational simulation.

In those examples we usually rely on coordinate attributes and/or classes that encapsulate xarray objects to implement the specific features that we need. While it works, it has limitations and I think it can be improved.

Custom coordinate wrappers would be a way of extending xarray that is very consistent with other current (or considered) extension mechanisms.

This is still a very vague idea and I’m sure that there are lots of details that can be discussed (serialization, etc.).

But before going further, I’d like to know your thoughts @pydata/xarray. Do you think it is a silly idea? Do you have in mind other use cases where custom coordinate wrappers would be useful?

The current implementation of the align function is problematic in the context of flexible indexes because:

• the sizes of the joined indexes are reused for checking compatibility with unlabelled dimension sizes
• the joined indexes are used as indexers to compute the aligned Dataset / DataArray.

This currently works well since a pd.Index can be directly treated as a 1-d array but this won’t be always the case anymore with custom indexes.

I'm opening this issue to gather ideas on how best to handle alignment in a more flexible way (I haven't been thinking much at this problem yet).

This adds PandasMultiIndexingAdapter._repr_html_ that can greatly speed-up the html repr of Xarray objects with multi-indexes.

This optimized _repr_html_ implementation is now used for formatting the array detailed view of all multi-index coordinates in the html repr, instead of converting the full index and each levels to numpy arrays before formatting them.

```python import xarray as xr

ds = xr.tutorial.load_dataset("air_temperature") da = ds["air"].stack(z=[...])

da.shape

(3869000,)

%timeit -n 1 -r 1 da.repr_html()

9.96 ms !

```

• [x] Closes #5529
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] Closes #6381
• [x] Tests added

• [x] Closes #4041
• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

This is not very elegant (actually plain text repr is already included in the notebook as text/plain mime type but it is ignored when text/html mime type is present), but it seems to work. I haven't found a better workaround.

I don't really know if this can be properly tested (I only added a basic test).

Steps to test this fix:

• To "untrust" a notebook: open an existing notebook with a simple editor, manually edit one output cell with a xarray object repr, and save the ipynb file.
• Open this notebook with the Notebook app, you should see the plain text repr.

This PR clears up the path for flexible indexes:

• it adds a new ~~IndexAdapter~~ Index base class that is meant to be inherited by all xarray-compatible indexes (built-in or 3rd-party)
• PandasIndexAdapter now inherits from ~~IndexAdapter~~ Index
• the xarray_obj.xindexes properties return Index (PandasIndexAdapter) instances. xarray_obj.indexes properties still return pandas.Index instances.

~~The latter is a breaking change, although I'm not sure if the indexes property has been made public yet.~~

This is still work in progress, there are many broken tests that are not fixed yet. (EDIT: all tests should be fixed now).

There's a lot of dirty fixes to avoid circular dependencies and in the many places where we still need direct access to the pandas.Index objects, but I'd expect that these will be cleaned-up further in the refactoring.

Xarray label-based data selection now relies on a newly added xarray.Index.query(self, labels: Dict[Hashable, Any]) -> Tuple[Any, Optional[None, Index]] method where:

• labels is a always a dictionary with coordinate name(s) as key(s) and the corresponding selection label(s) as values
• When calling .sel with some coordinate(s)/label(s) pairs, those are first grouped by index so that only the relevant pairs are passed to an Index.query
• the returned tuple contains the positional indexers and (optionally) a new index object

For a simple pd.Index, labels always corresponds to a 1-item dictionary like {'coord_name': label_values}, which is not very useful in this case, but this format is useful for pd.MultiIndex and will likely be for other, custom indexes.

Moving the label->positional indexer conversion logic into PandasIndex.query(), I've tried to separate pd.Index vs pd.MultiIndex concerns by adding a new PandasMultiIndex wrapper class (it will probably be useful for other things as well) and refactor the complex logic that was implemented in convert_label_indexer. Hopefully it is a bit clearer now.

Working towards a more flexible/generic system, we still need to figure out how to:

• pass index query extra arguments like method and tolerance for pd.Index but in a more generic way
• handle several positional indexers over multiple dimensions possibly returned by a custom "meta-index" (e.g., staggered grid index)
• handle the case of positional indexers returned from querying >1 indexes along the same dimension (e.g., multiple coordinates along x with a simple pd.Index)
• pandas indexes don't need information like the names or shapes of their corresponding coordinate(s) to perform label-based selection, but this kind of information will probably be needed for other indexes (we actually need it for advanced point-wise selection using tree-based indexes in xoak).

This could be done in follow-up PRs..

Side note: I've initially tried to return from xindexes items for multi-index levels as well (not only index dimensions), but it's probably wiser to save this for later (when we'll tackle the multi-index virtual coordinate refactoring) as there are many places in Xarray where this is clearly not expected.

Happy to hear your thoughts @pydata/xarray.

As a preliminary step before I take on the refactoring and implementation of flexible indexes in Xarray for the next few months, I reviewed the status of https://github.com/pydata/xarray/projects/1 and started compiling partially implemented or planned changes, thoughts, etc. into a single document that may serve as a basis for further discussion and implementation work.

It's still very much work in progress (I will update it regularly in the forthcoming days) and it is very open to discussion (we can use this PR for that)!

I'm not sure if Xarray's root folder is a good place for this document, though. We could move this into a new repository in xarray-contrib (that could also host other enhancement proposals) if that's necessary.

I'm looking forward to getting started on this and to getting your thoughts/feedback!

• [x] Closes #5384
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] Closes #6382
• [x] Tests added

• [x] Closes #6384
• [x] Tests added

• [x] Closes #6379
• [x] Tests added

• [x] Closes #6393
• [x] Tests added

• [x] Closes #6380
• [x] Tests added

• [x] Closes #6416
• [x] Tests added

When looking at #5691 I noticed that the labels are sometimes coerced to arrays (i.e., #3153) but not always.

Later in PandasIndex.query those may again be coerced to arrays (i.e., _as_array_tuplesafe). In #5692 (https://github.com/pydata/xarray/pull/5692/commits/a551c7f05abf90a492fb59068b59ebb2bac8cb4c) they are always coerced to arrays before maybe be converted as scalars.

Shouldn't we therefore make things easier and ensure that the labels given to xarray.Index.query() always have an array interface? This would also yield a more predictable behavior to anyone who wants to implement custom xarray indexes.

I guess it's rather an edge case, but a similar issue than the one fixed in #3153 may occur with multi-indexes:

```python

foo_data = ['a', 'a', 'b', 'b'] bar_data = np.array([0.1, 0.2, 0.7, 0.9], dtype=np.float32) da = xr.DataArray([1, 2, 3, 4], dims="x", coords={"foo": ("x", foo_data), "bar": ("x", bar_data)}) da = da.set_index(x=["foo", "bar"]) ```

```python

da.sel(bar=0.1) KeyError: 0.1 ```

```python

da.sel(bar=np.array(0.1, dtype=np.float32).item()) <xarray.DataArray (foo: 1)> array([1]) Coordinates: * foo (foo) object 'a' ```

(xarray version: 0.18.2 as there's a regression introduced in 0.19.0 #5691)

We should have some API in xarray.Index to update the index when its corresponding coordinate variables are renamed.

This currently implemented here where the underlying pd.Index name(s) are updated: https://github.com/pydata/xarray/blob/c5530d52d1bcbd071f4a22d471b728a4845ea36f/xarray/core/dataset.py#L3299-L3314

This logic should be moved into PandasIndex and PandasMultiIndex.

Other, custom indexes might also have internal attributes to update, so we might need formal API for that.

What happened:

Since #5569 I get a lot of mypy errors from _typed_ops.pyi (see below). What's weird is that it is not happening in all cases:

$ mypy # ok $ mypy . # errors $ pre-commit run --all-files # ok $ pre-commit run # errors $ git commit # (via pre-commit hooks) errors

I also tried pre-commit clean with no luck. EDIT: I also tried on a freshly cloned xarray repository.

@max-sixty @Illviljan Any idea on what's happening?

What you expected to happen:

No mypy error in all cases.

Anything else we need to know?:

xarray/core/_typed_ops.pyi:32: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:33: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:34: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:35: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:36: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:37: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:38: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:39: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:40: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:41: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:42: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:43: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:44: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:45: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:46: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:47: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:48: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:49: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:50: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:51: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:52: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:53: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:54: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:55: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:56: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:57: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:60: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:61: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:62: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:63: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:64: error: The erased type of self "xarray.core.dataset.Dataset" is not a supertype of its class "xarray.core._typed_ops.DatasetOpsMixin" [misc] xarray/core/_typed_ops.pyi:65: error: The erased type of self "xarray.core.dataset.Dataset" is not a supertype of its class "xarray.core._typed_ops.DatasetOpsMixin" [misc] xarray/core/_typed_ops.pyi:66: error: The erased type of self "xarray.core.dataset.Dataset" is not a supertype of its class "xarray.core._typed_ops.DatasetOpsMixin" [misc] xarray/core/_typed_ops.pyi:67: error: The erased type of self "xarray.core.dataset.Dataset" is not a supertype of its class "xarray.core._typed_ops.DatasetOpsMixin" [misc] xarray/core/_typed_ops.pyi:77: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:83: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:89: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:95: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:101: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:107: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:113: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:119: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:125: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:131: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:137: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:143: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:149: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:155: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:161: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:167: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:173: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:179: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:185: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:191: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:197: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:203: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:209: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:215: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:221: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:227: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:230: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:231: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:232: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:233: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:234: error: The erased type of self "xarray.core.dataarray.DataArray" is not a supertype of its class "xarray.core._typed_ops.DataArrayOpsMixin" [misc] xarray/core/_typed_ops.pyi:235: error: The erased type of self "xarray.core.dataarray.DataArray" is not a supertype of its class "xarray.core._typed_ops.DataArrayOpsMixin" [misc] xarray/core/_typed_ops.pyi:236: error: The erased type of self "xarray.core.dataarray.DataArray" is not a supertype of its class "xarray.core._typed_ops.DataArrayOpsMixin" [misc] xarray/core/_typed_ops.pyi:237: error: The erased type of self "xarray.core.dataarray.DataArray" is not a supertype of its class "xarray.core._typed_ops.DataArrayOpsMixin" [misc] xarray/core/_typed_ops.pyi:247: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:253: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:259: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:265: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:271: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:277: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:283: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:289: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:295: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:301: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:307: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:313: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:319: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:325: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:331: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:337: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:343: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:349: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:355: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:361: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:367: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:373: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:379: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:385: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:391: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:397: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:400: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:401: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:402: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:403: error: Self argument missing for a non-static method (or an invalid type for self) [misc] xarray/core/_typed_ops.pyi:404: error: The erased type of self "xarray.core.variable.Variable" is not a supertype of its class "xarray.core._typed_ops.VariableOpsMixin" [misc] xarray/core/_typed_ops.pyi:405: error: The erased type of self "xarray.core.variable.Variable" is not a supertype of its class "xarray.core._typed_ops.VariableOpsMixin" [misc] xarray/core/_typed_ops.pyi:406: error: The erased type of self "xarray.core.variable.Variable" is not a supertype of its class "xarray.core._typed_ops.VariableOpsMixin" [misc] xarray/core/_typed_ops.pyi:407: error: The erased type of self "xarray.core.variable.Variable" is not a supertype of its class "xarray.core._typed_ops.VariableOpsMixin" [misc]

Environment:

mypy 0.910 python 3.9.6 (also tested with 3.8)

Yesterday during the flexible indexes weekly meeting we have discussed with @shoyer and @jhamman on what would be the best approach to implement the new data model described here. In this issue I summarize the implementation of the current data model as well as some suggestions for the new data model along with their pros / cons (I might still be missing important ones!). I don't think there's an easy or ideal solution unfortunately, so @pydata/xarray any feedback would be very welcome!

Current data model implementation

Currently any (pandas) index is wrapped into an IndexVariable object through an intermediate adapter to preserve dtypes and handle explicit indexing. This allows directly reusing the index data as a xarray coordinate variable. For a pandas multi-index, virtual coordinates are created for each level from the IndexVariable object wrapping the index. Although relying on "virtual coordinates" more or less worked so far, it is over-complicated. Moreover, this wouldn't work with the new data model where an index may be built from a set of coordinates with different dimensions.

Proposed alternatives

Option 1: independent (coordinate) variables and indexes

Indexes and coordinates are loosely coupled, i.e., a xarray.Index holds a reference (mapping) to the coordinate variable(s) from which it is built but both manage their own data independently of each other.

Pros:

• separation of concerns.
• we don't need anymore those complicated adapters for reusing the index data as xarray (virtual) variable(s), which may simplify some xarray internals.
• if we drop an index, that's simple, we just drop it and all its related coordinate variables are left as-is.
• we could theoretically build a (pandas) index from a chunked coordinate, and then when we drop the index we still have this chunked coordinate left untouched.

Cons:

• data duplication
• this would clearly be a regression when using pandas indexes, but maybe less so for other indexes like kd-trees where adapting those objects for using it like coordinate variables wouldn't be easy or even possible.
• what if we want to build a DataArray or Dataset from one or more existing indexes (pandas or other)? Passing an index and treating as an array then re-building an index from this array is not optimal.
• keeping an index and its corresponding coordinate variable(s) in a consistent, in-sync state may be tricky, given that those variables may be mutable (although we could prevent this by encapsulating those variables using a very lightweight wrapper inspired by IndexVariable).

Option 2: indexes hold coordinate variables

This is the opposite approach of the current one. Here, a xarray.Index would wrap one or more xarray.Variable objects.

Pros:

• probably easier to keep an index and its corresponding coordinate variable(s) in-sync.
• sharing data between an index and its coordinate variables may be easier.

Cons:

• accessing / iterating through all coordinate variables in a DataArray or Dataset may be less straightforward.
• when the index is dropped, we might need some logic / API to return the coordinates as new xarray.Variable objects with their own data (or should we simply always drop the corresponding coordinates too? maybe not...).
• more responsibility / work for developers who want to provide 3rd party xarray indexes.

Option 3: intermediate solution

When an index is set (or unset), it returns a new set of coordinate variables to replace the existing ones.

Pros:

• it keeps some separation of concerns, while it allows data sharing through adapters and/or ensures that variables are immutable using lightweight wrappers.

Cons:

• like option 2, more things to care of for 3rd party xarray index developers.

EDIT: see https://github.com/pydata/xarray/issues/4118 for ongoing discussion

Probably it has been already suggested, but similarly to netCDF4 groups it would be nice if we could access Dataset data variables, coordinates and attributes via groups.

Currently xarray allows loading a specific netCDF4 group into a Dataset. Different groups can be loaded as separate Dataset objects, which may be then combined into a single, flat Dataset. Yet, in some cases it makes sense to represent data as a single object while it would be convenient to keep some nested structure. For example, a Dataset representing data on a staggered grid might have scalar_vars and flux_vars groups. Here are some potential uses for groups. When there are a lot of data variables and/or attributes, it would also help to have a more concise repr.

I think about an implementation of Dataset.groups that would be specific to xarray, i.e., independent of any backend, and which would easily co-exist with the flat Dataset. It shouldn't be required for a backend to support groups (some existing backends simply don't). It is up to each backend to eventually transpose the Dataset.groups logic to its own group logic.

Dataset.groups might return a DatasetGroups object, which quite similarly to xarray.core.coordinates.DatasetCoordinates would (1) have a reference to the Dataset object, (2) basically consist of a Mapping of group names to data variable/coordinate/attribute names and (3) dynamically create another Dataset object (sub-dataset) on __getitem__. Keys of Dataset.groups should be accessible as attributes , e.g., ds.groups['scalar_vars'] == ds.scalar_vars.

Questions:

• How to handle hierarchies of > 1 levels (i.e., groups of groups...)?
• How to ensure that a variable / attribute in one group is not also present in another group?
• Case of methods called from groups with inplace=True?

• ~~Closes #xxxx~~
• [x] Tests added / passed
• [x] Passes git diff upstream/master | flake8 --diff
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

~~In addition to Dataset repr, the error message also shows the output of Dataset.info() for both datasets.~~

~~This may not be the most elegant solution, but it is helpful when datasets only differ by their attributes attached to coordinates or data variables (not shown in repr). I'm open to any suggestion.~~

The report shows the differences for dimensions, data values (Variable and DataArray), coordinates, data variables and attributes (the latter only for testing.assert_identical).

There is currently not much tests for xarray.testing functions, but I'm willing to add more if needed.

Not sure if it's worth a what's new entry (EDIT: added one).

• [x] Closes #1722
• [x] Tests added / passed
• [x] Passes git diff upstream/master **/*py | flake8 --diff
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [x] Closes #1627
• [ ] Tests added
• [ ] Tests passed
• [ ] Passes git diff upstream/master **/*py | flake8 --diff
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

This is work in progress, although the basic functionality is there. You can see a preview here: http://nbviewer.jupyter.org/gist/benbovy/3009f342fb283bd0288125a1f7883ef2

TODO:

• [ ] Add support for Multi-indexes
• [ ] Probably good to have some opt-in or fail back system in case where we (or users) know that the rendering will not work
• [ ] Add some tests

Nice to have (keep this for later):

• Clean-up CSS code and HTML template (track CSS subgrid support in browsers, this may simplify a lot the things here).
• Dynamically adapt cell widths (given the length of the names of variables and dimensions). Currently all cells have a fixed width. This is tricky, though, as we don't use a monospace font here.
• Integration with jupyterlab/notebook themes (CSS classes) and maybe allow custom CSS.
• Integration of Dask arrays HTML repr (+ integration of repr for other array backends).
• Maybe find a way (if possible) to include CSS only once in the notebook (currently it is included each time a xarray object is displayed in an output cell, which is not very nice).
• Review the rules for collapsing the Coordinates, Data variables and Attributes sections (maybe expose them as global options).
• Maybe also define some rules to collapse automatically the data section (DataArray and Variable) when the data repr is too long.
• Maybe add rich representation for Dataset.coords and Dataset.data_vars as well?

Edit: preview for Dataset and DataArray (pure html/css)

Dataset: https://jsfiddle.net/tay08cn9/4/ DataArray: https://jsfiddle.net/43z4v2wt/9/

I started to think a bit more deeply about how could look like a more rich, html-based representation of xarray objects that we would see, e.g., in jupyter notebooks.

Here are some ideas for Dataset: https://jsfiddle.net/9ab4c3tr/35/

Some notes: - The html repr looks pretty similar than the plain-text repr. I think it's better if they don't differ too much from each other. - For the sake of consistency, I've stolen some style from pandas.Dataframe repr as it is shown in jupyterlab. - I tried to emphasize the most important parts of the repr, i.e., the lists of dimensions, coordinates and variables. - I think it's best if we keep a very lightweight implementation, i.e., pure HTML/CSS (no Javascript). It already allows some interaction like hover effects and collapsible sections. However, I doubt that more fancy stuff (like, e.g., highlighting on hover a specific dimension simultaneously at several places of the repr) would be possible here without Javascript. I have limited skills in this area, though.

It is still, of course, some preliminary thoughts. Any feedback/suggestion is welcome, even opinions about whether an html repr is really needed or not!

I'm wondering what would be a good name for a package that primarily aims at providing an xarray extension (in the form of a DataArray and/or Dataset accessor).

I'm currently thinking about using a prefix like the scikit package family (e.g., scikit-learn, scikit-image).

For example, for a xarray extension for signal processing we would have:

package full name: xarray-signal package import name: xrsignal (like sklearn) accessor name: signal.

```python

import xarray as xr import xrsignal ds = xr.Dataset() ds.signal.process(...) ```

The main advantage is that we directly have an idea on what the package is about. It may be also good for the overall visibility of both xarray and its 3rd-party extensions. The downside is that there is three name variations: one for getting and installing the package, another one for importing the package and again another one for using the accessor. This may be annoying especially for new users who are not accustomed to this kind of naming convention.

Conversely, choosing a different, unrelated name like salem or pangaea has the advantage of using the same name everywhere and perhaps providing multiple accessors in the same package, but given that the number of xarray extensions is likely to grow in a next future (see, e.g., the pangeo-data project) it would become difficult to have a clear view of the whole xarray package ecosystem.

Any thoughts?

I often have to deal with datasets in this form (multiple variables of different sizes, each representing different categories, on the same physical dimension but using different names as they have different labels),

<xarray.Dataset> Dimensions: (wn_band1: 4, wn_band2: 6, wn_band3: 8) Coordinates: * wn_band1 (wn_band1) float64 200.0 266.7 333.3 400.0 * wn_band2 (wn_band2) float64 500.0 560.0 620.0 680.0 740.0 800.0 * wn_band3 (wn_band3) float64 1.5e+03 1.643e+03 1.786e+03 1.929e+03 ... Data variables: data_band3 (wn_band3) float64 0.7515 0.5302 0.6697 0.9621 0.01815 ... data_band1 (wn_band1) float64 0.3801 0.6649 0.01884 0.9407 data_band2 (wn_band2) float64 0.8813 0.4481 0.2353 0.9681 0.1085 0.0835

where it would be more convenient to have the data re-arranged into the following form (concatenate the variables into a single variable with a multi-index with the labels of both the categories and the physical coordinate):

<xarray.Dataset> Dimensions: (spectrum: 18) Coordinates: * spectrum (spectrum) MultiIndex - band (spectrum) int64 1 1 1 1 2 2 2 2 2 2 3 3 3 3 3 3 3 3 - wn (spectrum) float64 200.0 266.7 333.3 400.0 500.0 560.0 620.0 ... Data variables: data (spectrum) float64 0.3801 0.6649 0.01884 0.9407 0.8813 0.4481 ...

The latter would allow using xarray's nice features like ds.groupby('band').mean().

Currently, the best way that I've found to transform the data is something like:

``` python data = np.concatenate([ds.data_band1, ds.data_band2, ds.data_band3]) wn = np.concatenate([ds.wn_band1, ds.wn_band2, ds.wn_band3]) band = np.concatenate([np.repeat(1, 4), np.repeat(2, 6), np.repeat(3, 8)])

midx = pd.MultiIndex.from_arrays([band, wn], names=('band', 'wn')) ds2 = xr.Dataset({'data': ('spectrum', data)}, coords={'spectrum': midx}) ```

Maybe I miss a better way to do this? If I don't, it would be nice to have a convenience method for this, unless this use case is too rare to be worth it. Also not sure at all on what would be a good API such a method.

Make xarray-related projects more discoverable, as it has been suggested in xarray mailing-list.

Not a big change, but adds a little more clarity IMO.

I'm open to any suggestion for better section names and/or organization. Also I let "What's new" at the top, but not sure if "Getting started" is the right section.

I create a big netcdf file like this:

```python In [1]: import xarray as xr

In [2]: import numpy as np

In [3]: ds = xr.Dataset({'myvar': np.arange(100000000, dtype='float64')})

In [4]: ds.to_netcdf('test.nc')

```

Then when I open the file in a IPython console and I use auto-completion, it triggers loading the data.

```python In [1]: import xarray as xr

In [2]: ds = xr.open_dataset('test.nc')

In [3]: ds.my # <TAB> autocompletion with any character -> triggers loading ```

I don't have that issue using the python console. Auto-completion for dictionary access in IPython (#1632) works fine too.

Output of xr.show_versions()

I use xarray 0.9.6 for both examples below.

With pandas 0.20.3, Dataset.set_index gives me what I expect (i.e., the grid__x data variable becomes a coordinate x):

```python In [1]: import xarray as xr

In [2]: import pandas as pd

In [3]: pd.version Out[3]: '0.20.3'

In [4]: ds = xr.Dataset({'grid__x': ('x', [1, 2, 3])})

In [5]: ds.set_index(x='grid__x') Out[5]: <xarray.Dataset> Dimensions: (x: 3) Coordinates: * x (x) int64 1 2 3 Data variables: empty ```

With pandas 0.21.0, it creates a MultiIndex, which is not what I expect here when setting an index with only one data variable:

```python In [1]: import xarray as xr

In [2]: import pandas as pd

In [3]: pd.version Out[3]: '0.21.0'

In [4]: ds = xr.Dataset({'grid__x': ('x', [1, 2, 3])})

In [5]: ds.set_index(x='grid__x') Out[5]: <xarray.Dataset> Dimensions: (x: 3) Coordinates: * x (x) MultiIndex - grid__x (x) int64 1 2 3 Data variables: empty ```

• [x] Closes #1303
• [x] Tests added / passed
• [x] Passes git diff upstream/master | flake8 --diff (if we ignore messages for .rst files and "imported but not used" messages for xarray.__init__.py)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Make xarray.core.variable.as_variable part of the public API and accessible as a top-level function: xarray.as_variable.

I changed the docstrings to follow the numpydoc format more closely.

I also removed the copy=False keyword arguments as apparently it was unused.

[Edited for more clarity]

First of all, I find the MultiIndex very useful and I'm looking forward to see the TODOs in #719 implemented in the next releases, especially the three first ones in the list!

Apart from these issues, I think that some other aspects may be improved, notably regarding data selection. Or maybe I've not correctly understood how to deal with multi-index and data selection...

To illustrate this, I use some fake spectral data with two discontinuous bands of different length / resolution:

``` In [1]: import pandas as pd

In [2]: import xarray as xr

In [3]: band = np.array(['foo', 'foo', 'bar', 'bar', 'bar'])

In [4]: wavenumber = np.array([4050.2, 4050.3, 4100.1, 4100.3, 4100.5])

In [5]: spectrum = np.array([1.7e-4, 1.4e-4, 1.2e-4, 1.0e-4, 8.5e-5])

In [6]: s = pd.Series(spectrum, index=[band, wavenumber])

In [7]: s.index.names = ('band', 'wavenumber')

In [8]: da = xr.DataArray(s, dims='band_wavenumber')

In [9]: da Out[9]: <xarray.DataArray (band_wavenumber: 5)> array([ 1.70000000e-04, 1.40000000e-04, 1.20000000e-04, 1.00000000e-04, 8.50000000e-05]) Coordinates: * band_wavenumber (band_wavenumber) object ('foo', 4050.2) ... ```

I extract the band 'bar' using sel:

``` In [10]: da_bar = da.sel(band_wavenumber='bar')

In [11]: da_bar Out[11]: <xarray.DataArray (band_wavenumber: 3)> array([ 1.20000000e-04, 1.00000000e-04, 8.50000000e-05]) Coordinates: * band_wavenumber (band_wavenumber) object ('bar', 4100.1) ... ```

It selects the data the way I want, although using the dimension name is confusing in this case. It would be nice if we can also use the MultiIndex names as arguments of the sel method, even though I don't know if it is easy to implement.

Futhermore, da_bar still has the 'band_wavenumber' dimension and the 'band' index-level, but it is not very useful anymore. Ideally, I'd rather like to obtain a DataArray object with a 'wavenumber' dimension / coordinate and the 'bar' band name dropped from the multi-index, i.e., something would require automatic index-level removal and/or automatic unstack when selecting data.

Extracting the band 'bar' from the pandas Series object gives something closer to what I need (see below), but using pandas is not an option as my spectral data involves other dimensions (e.g., time, scans, iterations...) not shown here for simplicity.

``` In [12]: s_bar = s.loc['bar']

In [13]: s_bar Out[13]: wavenumber 4100.1 0.000120 4100.3 0.000100 4100.5 0.000085 dtype: float64 ```

The problem is also that the unstacked DataArray object resulting from the selection has the same dimensions and size than the original, unstacked DataArray object. The only difference is that unselected values are replaced by nan.

``` In [13]: da.unstack('band_wavenumber') Out[13]: <xarray.DataArray (band: 2, wavenumber: 5)> array([[ nan, nan, 1.20000000e-04, 1.00000000e-04, 8.50000000e-05], [ 1.70000000e-04, 1.40000000e-04, nan, nan, nan]]) Coordinates: * band (band) object 'bar' 'foo' * wavenumber (wavenumber) float64 4.05e+03 4.05e+03 4.1e+03 4.1e+03 4.1e+03

In [14]: da_bar.unstack('band_wavenumber') Out[14]: <xarray.DataArray (band: 2, wavenumber: 5)> array([[ nan, nan, 1.20000000e-04, 1.00000000e-04, 8.50000000e-05], [ nan, nan, nan, nan, nan]]) Coordinates: * band (band) object 'bar' 'foo' * wavenumber (wavenumber) float64 4.05e+03 4.05e+03 4.1e+03 4.1e+03 4.1e+03 ```

Implements 2, 4 and 5 in #719.

Demo:

``` In [1]: import numpy as np

In [2]: import pandas as pd

In [3]: import xarray as xr

In [4]: index = pd.MultiIndex.from_product((list('ab'), range(2)), ...: names= ('level_1', 'level_2'))

In [5]: da = xr.DataArray(np.random.rand(4, 4), coords={'x': index}, ...: dims=('x', 'y'), name='test')

In [6]: da Out[6]: <xarray.DataArray 'test' (x: 4, y: 4)> array([[ 0.15036153, 0.68974802, 0.40082234, 0.94451318], [ 0.26732938, 0.49598123, 0.8679231 , 0.6149102 ], [ 0.3313594 , 0.93857424, 0.73023367, 0.44069622], [ 0.81304837, 0.81244159, 0.37274953, 0.86405196]]) Coordinates: * level_1 (x) object 'a' 'a' 'b' 'b' * level_2 (x) int64 0 1 0 1 * y (y) int64 0 1 2 3

In [7]: da['level_1'] Out[7]: <xarray.DataArray 'level_1' (x: 4)> array(['a', 'a', 'b', 'b'], dtype=object) Coordinates: * level_1 (x) object 'a' 'a' 'b' 'b' * level_2 (x) int64 0 1 0 1

In [8]: da.sel(x='a', level_2=1) Out[8]: <xarray.DataArray 'test' (y: 4)> array([ 0.26732938, 0.49598123, 0.8679231 , 0.6149102 ]) Coordinates: x object ('a', 1) * y (y) int64 0 1 2 3

In [9]: da.sel(level_2=1) Out[9]: <xarray.DataArray 'test' (level_1: 2, y: 4)> array([[ 0.26732938, 0.49598123, 0.8679231 , 0.6149102 ], [ 0.81304837, 0.81244159, 0.37274953, 0.86405196]]) Coordinates: * level_1 (level_1) object 'a' 'b' * y (y) int64 0 1 2 3 ```

Some notes about the implementation: - I slightly modified Coordinate so that it allows setting different values for the names of the coordinate and its dimension. There is no breaking change. - I also added a Coordinate.get_level_coords method to get independent, single-index coordinates objects from a MultiIndex coordinate.

Remaining issues: - Coordinate.get_level_coords calls pandas.MultiIndex.get_level_values for each level and is itself called each time when indexing and for repr. This can be very costly!! It would be nice to return some kind of lazy index object instead of computing the actual level values. - repr replace a MultiIndex coordinate by its level coordinates. That can be confusing in some cases (see below). Maybe we can set a different marker than * for level coordinates.

``` In [6]: [name for name in da.coords] Out[6]: ['x', 'y']

In [7]: da.coords.keys() Out[7]: KeysView(Coordinates: * level_1 (x) object 'a' 'a' 'b' 'b' * level_2 (x) int64 0 1 0 1 * y (y) int64 0 1 2 3) `` -DataArray.level_1doesn't return anotherDataArray` object:

In [10]: da.level_1 Out[10]: <xarray.Coordinate 'level_1' (x: 4)> array(['a', 'a', 'b', 'b'], dtype=object) - Maybe we need to test the uniqueness of level names at DataArray or Dataset creation.

Of course still needs proper tests and docs...

Another item of #719.

An example:

``` python

index = pd.MultiIndex.from_product((list('ab'), range(10))) index.names= ('a_long_level_name', 'level_1') data = xr.DataArray(range(20), [('x', index)]) data <xarray.DataArray (x: 20)> array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]) Coordinates: * x (x) object MultiIndex - a_long_level_name object 'a' 'a' 'a' 'a' 'a' 'a' 'a' 'a' 'a' 'a' 'b' ... - level_1 int64 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 ```

To be consistent with the displayed coordinates and/or data variables, it displays the actual used level values. Using the pandas.MultiIndex.get_level_values method would be expensive for big indexes, so I re-implemented it in xarray so that we can truncate the computation to the first x values, which is very cheap.

It still needs testing.

Maybe it would be nice to align the level values.

Given #844 and #916, maybe it might be useful to also have a Dataset.filter_by_name method?

I currently deal with datasets that have many data variables with names like:

... reference__HONO (rlevel) float64 3.16e-15 1e-14 1e-14 1e-14 ... reference__NO (rlevel) float64 2.16e-05 3.57e-06 9.3e-07 ... reference__HO2NO2 (rlevel) float64 9.58e-20 7.32e-19 4.63e-18 ... ... retrieved__O3 (level) float64 1.552e-06 5.618e-07 ... retrieved__N2O (level) float64 4.714e-11 9.905e-11 ... retrieved__CO2 (level) float64 0.0002816 0.0003592 ... ...

Using ds.filter_by_name(like='reference__') would be less verbose than, e.g., xr.Dataset({name: ds[name] for name in ds.keys() if 'reference__' in name}), unless there is already a more convenient way that I'm missing?

Follows #767.

This is incomplete (it still needs some tests and documentation updates), but it is working for both Dataset and DataArray objects. I also don't know if it is fully compatible with lazy indexing (Dask).

Using the example from #767:

In [4]: da.sel(band_wavenumber={'band': 'foo'}) Out[4]: <xarray.DataArray (wavenumber: 2)> array([ 0.00017, 0.00014]) Coordinates: * wavenumber (wavenumber) float64 4.05e+03 4.05e+03

As shown in this example, similarily to pandas, it automatically renames the dimension and assigns a new coordinate when the selection doesn't return a pd.MultiIndex (here it returns a pd.FloatIndex).

In some cases this behavior may be unwanted (??), so I added a drop_level keyword argument (if False it keeps the multi-index and doesn't change the dimension/coordinate names):

In [5]: da.sel(band_wavenumber={'band': 'foo'}, drop_level=False) Out[5]: <xarray.DataArray (band_wavenumber: 2)> array([ 0.00017, 0.00014]) Coordinates: * band_wavenumber (band_wavenumber) object ('foo', 4050.2) ('foo', 4050.3)

Note that it also works with DataArray.loc, but (for now) in that case it always returns the multi-index:

In [6]: da.loc[{'band_wavenumber': {'band': 'foo'}}] Out[6]: <xarray.DataArray (band_wavenumber: 2)> array([ 0.00017, 0.00014]) Coordinates: * band_wavenumber (band_wavenumber) object ('foo', 4050.2) ('foo', 4050.3)

This is however inconsistent with Dataset.sel and Dataset.loc that both apply drop_level=True by default, due to their different implementation. Two solutions: (1) make DataArray.loc apply drop_level by default, or (2) use drop_level=False by default everywhere.

Fixes #769.