1. This moves the region detection code into ZarrStore so we only open the store once.
2. Instead of opening the store as a dataset, construct a pd.Index directly to "auto"-infer the region.

The diff is large mostly because a bunch of code moved from backends/api.py to backends/zarr.py

Builds on #8997

@andersy005 I think it would be good to keep a running list of NamedArray tasks. I'll start with a rough sketch, please update/edit as you like.

• [x] Refactor out NamedArray base class (#8075)
• [x] publicize design doc: Scientific Python | Pangeo | NumPy Mailist
• [ ] Migrate VariableArithmetic to NamedArrayArithmetic (#8244)
• [ ] Migrate ExplicitlyIndexed array classes to array protocols
• [x] MIgrate from *Indexer objects to .oindex and .vindex on ExplicitlyIndexed array classes
• [ ] https://github.com/pydata/xarray/pull/8870
• [ ] Migrate unary ops
• [ ] Migrate binary ops
• [ ] Migrate nanops.py
• [x] Avoid "injecting" reduce methods potentially by using generate_reductions.py? (#8304)
• [ ] reprs and formatting.py
• [x] parallelcompat.py
• [ ] pycompat.py (#8244)
• [ ] https://github.com/pydata/xarray/pull/8276
• [ ] have test_variable.py test both NamedArray and Variable
• [x] Arrays with unknown shape #8291
• [ ] https://github.com/pydata/xarray/issues/8306
• [ ] https://github.com/pydata/xarray/issues/8310
• [ ] https://github.com/pydata/xarray/issues/8333
• [ ] Try to preserve imports from xarray.core/* by importing namedarray functionality into xarray.core/*

xref #3981

Is your feature request related to a problem?

Today if users have to concurrently load multiple variables in a DataArray or Dataset, they have to use dask.

It struck me that it'd be pretty easy for .load to gain an executor kwarg that accepts anything that follows the concurrent.futures executor interface, and parallelize this loop.

https://github.com/pydata/xarray/blob/b0036749542145794244dee4c4869f3750ff2dee/xarray/core/dataset.py#L853-L857

1. Now that numba supports 3.12.
2. Disabled pint on the main environment since it doesn't work. Pint is still installed in the all-but-dask env, which is still runs python 3.11 for this reason.

What is your issue?

cc @hmaarrfk @kmuehlbauer

Passed: https://github.com/pydata/xarray/actions/runs/4115923717/jobs/7105298426 Failed: https://github.com/pydata/xarray/actions/runs/4115946392/jobs/7105345290

Versions: h5netcdf 1.1.0 pyhd8ed1ab_0 conda-forge h5py 3.8.0 nompi_py310h5555e59_100 conda-forge hdf4 4.2.15 h7aa5921_5 conda-forge hdf5 1.12.2 nompi_h48135f9_101 conda-forge

``` =================================== FAILURES =================================== ___ test_open_mfdataset_manyfiles[h5netcdf-20-True-5-5] ______ [gw1] darwin -- Python 3.10.9 /Users/runner/micromamba-root/envs/xarray-tests/bin/python

readengine = 'h5netcdf', nfiles = 20, parallel = True, chunks = 5 file_cache_maxsize = 5

@requires_dask
@pytest.mark.filterwarnings("ignore:use make_scale(name) instead")
def test_open_mfdataset_manyfiles(
    readengine, nfiles, parallel, chunks, file_cache_maxsize
):
    # skip certain combinations
    skip_if_not_engine(readengine)

    if ON_WINDOWS:
        pytest.skip("Skipping on Windows")

    randdata = np.random.randn(nfiles)
    original = Dataset({"foo": ("x", randdata)})
    # test standard open_mfdataset approach with too many files
    with create_tmp_files(nfiles) as tmpfiles:
        writeengine = readengine if readengine != "pynio" else "netcdf4"
        # split into multiple sets of temp files
        for ii in original.x.values:
            subds = original.isel(x=slice(ii, ii + 1))
            if writeengine != "zarr":
                subds.to_netcdf(tmpfiles[ii], engine=writeengine)
            else:  # if writeengine == "zarr":
                subds.to_zarr(store=tmpfiles[ii])

        # check that calculation on opened datasets works properly

      with open_mfdataset(
            tmpfiles,
            combine="nested",
            concat_dim="x",
            engine=readengine,
            parallel=parallel,
            chunks=chunks if (not chunks and readengine != "zarr") else "auto",
        ) as actual:

/Users/runner/work/xarray/xarray/xarray/tests/test_backends.py:3267:

/Users/runner/work/xarray/xarray/xarray/backends/api.py:991: in open_mfdataset datasets, closers = dask.compute(datasets, closers) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/base.py:599: in compute results = schedule(dsk, keys, kwargs) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/threaded.py:89: in get results = get_async( /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py:511: in get_async raise_exception(exc, tb) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py:319: in reraise raise exc /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py:224: in execute_task result = _execute_task(task, data) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/core.py:119: in _execute_task return func((_execute_task(a, cache) for a in args)) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/utils.py:72: in apply return func(args, kwargs) /Users/runner/work/xarray/xarray/xarray/backends/api.py:526: in open_dataset backend_ds = backend.open_dataset( /Users/runner/work/xarray/xarray/xarray/backends/h5netcdf_.py:417: in open_dataset ds = store_entrypoint.open_dataset( /Users/runner/work/xarray/xarray/xarray/backends/store.py:32: in open_dataset vars, attrs = store.load() /Users/runner/work/xarray/xarray/xarray/backends/common.py:129: in load (decode_variable_name(k), v) for k, v in self.get_variables().items() /Users/runner/work/xarray/xarray/xarray/backends/h5netcdf.py:220: in get_variables return FrozenDict( /Users/runner/work/xarray/xarray/xarray/core/utils.py:471: in FrozenDict return Frozen(dict(args, *kwargs)) /Users/runner/work/xarray/xarray/xarray/backends/h5netcdf_.py:221: in <genexpr> (k, self.open_store_variable(k, v)) for k, v in self.ds.variables.items() /Users/runner/work/xarray/xarray/xarray/backends/h5netcdf_.py:200: in open_store_variable elif var.compression is not None: /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/h5netcdf/core.py:394: in compression return self._h5ds.compression

self = <[AttributeError("'NoneType' object has no attribute '_root'") raised in repr()] Variable object at 0x151378970>

@property
def _h5ds(self):
    # Always refer to the root file and store not h5py object
    # subclasses:

  return self._root._h5file[self._h5path]

E AttributeError: 'NoneType' object has no attribute '_h5file'

```

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Expose Grouper and Resampler as public API

TODO: - [ ] Consider avoiding IndexVariable

• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

What happened?

From https://github.com/pydata/xarray/discussions/8951

if d is a MultiIndex-ed dataset with levels (x, y, z), and m is a dataset with a single coord x m.isel(x=d.x) builds a dataset with a MultiIndex with levels (y, z). This seems like it should work.

cc @benbovy

What did you expect to happen?

No response

Minimal Complete Verifiable Example

```Python import pandas as pd, xarray as xr, numpy as np

xr.set_options(use_flox=True)

test = pd.DataFrame() test["x"] = np.arange(100) % 10 test["y"] = np.arange(100) test["z"] = np.arange(100) test["v"] = np.arange(100)

d = xr.Dataset.from_dataframe(test) d = d.set_index(index = ["x", "y", "z"]) print(d)

m = d.groupby("x").mean() print(m)

print(d.xindexes) print(m.isel(x=d.x).xindexes)

xr.align(d, m.isel(x=d.x))

res = d.groupby("x") - m

print(res)

```

<xarray.Dataset> Dimensions: (index: 100) Coordinates: * index (index) object MultiIndex * x (index) int64 0 1 2 3 4 5 6 7 8 9 0 1 2 ... 8 9 0 1 2 3 4 5 6 7 8 9 * y (index) int64 0 1 2 3 4 5 6 7 8 9 ... 90 91 92 93 94 95 96 97 98 99 * z (index) int64 0 1 2 3 4 5 6 7 8 9 ... 90 91 92 93 94 95 96 97 98 99 Data variables: v (index) int64 0 1 2 3 4 5 6 7 8 9 ... 90 91 92 93 94 95 96 97 98 99 <xarray.Dataset> Dimensions: (x: 10) Coordinates: * x (x) int64 0 1 2 3 4 5 6 7 8 9 Data variables: v (x) float64 45.0 46.0 47.0 48.0 49.0 50.0 51.0 52.0 53.0 54.0 Indexes: ┌ index PandasMultiIndex │ x │ y └ z Indexes: ┌ index PandasMultiIndex │ y └ z ValueError...

MVCE confirmation

• [x] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [x] Complete example — the example is self-contained, including all data and the text of any traceback.
• [x] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [x] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [x] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Update the cache always.

What happened?

Found by hypothesis ``` import xarray as xr import numpy as np

var = xr.Variable(dims="2", data=np.array(['1970-01-01T00:00:00.000000000', '1970-01-01T00:00:00.000000002', '1970-01-01T00:00:00.000000001'], dtype='datetime64[ns]')) var1 = xr.Variable(data=np.array([0], dtype=np.uint32), dims=['1'], attrs={})

state = xr.Dataset() state['2'] = var state = state.stack({"0": ["2"]}) state['1'] = var1 state['1_'] = var1#.copy(deep=True) state = state.swap_dims({"1": "1_"}) xr.testing.assertions._assert_internal_invariants(state, False) ```

This swaps simple pandas indexed dims, but the multi-index that is in the dataset and not affected by the swap_dims op ends up broken.

cc @benbovy

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [ ] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

@benbovy this seems to be the root cause of #8646, the variable name in Dataset._variables does not match IndexVariable.name.

A good number of tests seem to fail though, so not sure if this is a good chck.

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Discussed in https://github.com/pydata/xarray/discussions/8751

• [x] Closes #8753

This seems to have been around since 2016-ish, so presumably our backend code path is passing arrays around, not Variables.

cc @ilan-gold

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

I was curious to see if the hypothesis stateful testing would catch an inconsistent sequence of index manipulation operations like #8646. Turns out rename_vars is basically broken? (filed #8659) :P

PS: this blog post is amazing. E state = DatasetStateMachine() E state.assert_invariants() E > === E E <xarray.Dataset> E Dimensions: () E Data variables: E *empty* E === E E E > vars: ('1', '1_') E state.add_dim_coord(var=<xarray.Variable (1: 1)> E array([0], dtype=uint32)) E state.assert_invariants() E > === E E <xarray.Dataset> E Dimensions: (1: 1) E Coordinates: E * 1 (1) uint32 0 E Data variables: E 1_ (1) uint32 0 E === E E E > renaming 1 to 0 E state.rename_vars(newname='0') E state.assert_invariants() E > === E E <xarray.Dataset> E Dimensions: (1: 1) E Coordinates: E * 0 (1) uint32 0 E Dimensions without coordinates: 1 E Data variables: E 1_ (1) uint32 0 E === E E E state.teardown()

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

joining the party - [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Closes #8252 Closes #3921

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Toward #8510, builds on #8776

This adds a proposal for decoupling the lazy indexing array machinery, indexing adapter machinery, and Variable's setitem and getitem methods, so that the latter can be migrated to NamedArray.

cc @andersy005

• [x] xref #8844

• [x] Closes #8343
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

The intended pattern is: ```python

after_init = initialize_zarr(store, ds, region_dims=("x",))
for i in range(ds.sizes["x"]):
    after_init.isel(x=[i]).to_zarr(store, region={"x": slice(i, i + 1)})

```

cc @slevang

We need to read existing variables to make sure we append or write to a region with the right encoding. Currently we decode all arrays in a Zarr group. Instead only decode those arrays for which we require encoding information.

What happened?

Resetting a MultiIndex dim coordinate preserves the MultiIndex levels as IndexVariables. We should either reset the indexes for the multiindex level variables, or warn asking the users to do so

This seems to be the root cause exposed by https://github.com/pydata/xarray/pull/8809

cc @benbovy

What did you expect to happen?

No response

Minimal Complete Verifiable Example

```Python import numpy as np import xarray as xr

ND DataArray that gets stacked along a multiindex

da = xr.DataArray(np.ones((3, 3)), coords={"dim1": [1, 2, 3], "dim2": [4, 5, 6]}) da = da.stack(feature=["dim1", "dim2"])

Extract just the stacked coordinates for saving in a dataset

ds = xr.Dataset(data_vars={"feature": da.feature}) xr.testing.assertions._assert_internal_invariants(ds.reset_index(["feature", "dim1", "dim2"]), check_default_indexes=False) # succeeds xr.testing.assertions._assert_internal_invariants(ds.reset_index(["feature"]), check_default_indexes=False) # fails, but no warning either ```

What happened?

Currently we don't overwrite indexes when region="auto" https://github.com/pydata/xarray/blob/e6ccedb56ed4bc8d0b7c1f16ab325795330fb19a/xarray/backends/api.py#L1769-L1770

I propose we do this for all region writes and completely disallow modifying indexes with a region write.

This would match the map_blocks model, where all indexes are specified in the template and no changes by the mapped function are allowed.

• [x] Closes #8589
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

cc @slevang

What is your issue?

Our choice to enable bottleneck by default results in quite a few issues about numerical stability and funny dtype behaviour: #7336, #7128, #2370, #1346 (and probably more)

Shall we disable it by default?

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

@aulemahal would you be able to test against xclim's test suite. I imagine you're doing a bunch of grouped quantiles.

…roadcast=False`` (#8784)"

This reverts commit 11f89ecdd41226cf93da8d1e720d2710849cd23e.

Reverting #8784

Sadly that PR broke a lot of tests by breaking create_test_data with from xarray.tests import create_test_data create_test_data()

```

AssertionError Traceback (most recent call last) Cell In[3], line 2 1 from xarray.tests import create_test_data ----> 2 create_test_data()

File ~/repos/xarray/xarray/tests/init.py:329, in create_test_data(seed, add_attrs, dim_sizes) 327 obj.coords["numbers"] = ("dim3", numbers_values) 328 obj.encoding = {"foo": "bar"} --> 329 assert all(var.values.flags.writeable for var in obj.variables.values()) 330 return obj

AssertionError: ```

Somehow that code changes whether IndexVariable.values returns a writeable numpy array. I spent some time debugging but couldn't figure it out.

cc @etienneschalk

xref #8844 Closes https://github.com/xarray-contrib/flox/issues/342

• [x] Closes #8843
• [x] Tests added

```python import numpy as np import pandas as pd

pd.set_option("mode.copy_on_write", True)

from xarray.core.variable import _possibly_convert_objects

string_var = np.array(["a", "bc", "def"], dtype=object) datetime_var = np.array( ["2019-01-01", "2019-01-02", "2019-01-03"], dtype="datetime64[ns]" ) assert _possibly_convert_objects(string_var).flags.writeable assert _possibly_convert_objects(datetime_var).flags.writeable ```

The core issue is that we now get read-only arrays back from pandas here: https://github.com/pydata/xarray/blob/fbcac7611bf9a16750678f93483d3dbe0e261a0a/xarray/core/variable.py#L197-L212

@phofl is this expected?

What is your issue?

This line fails with pd.set_options("mode.copy_on_write", True) https://github.com/pydata/xarray/blob/c9d3084e98d38a7a9488380789a8d0acfde3256f/xarray/tests/init.py#L329

We'll need to fix this before Pandas 3 is released in April: https://github.com/pydata/xarray/blob/c9d3084e98d38a7a9488380789a8d0acfde3256f/xarray/tests/init.py#L329

Here's a test ```python def example(): obj = Dataset() obj["dim2"] = ("dim2", 0.5 * np.arange(9)) obj["time"] = ("time", pd.date_range("2000-01-01", periods=20) print({k: v.data.flags for k, v in obj.variables.items()}) return obj

example() pd.set_options("mode.copy_on_write", True) example() ```

What happened?

(xref #8658)

I'm not sure what the expected behaviour is here:

```python import xarray as xr import numpy as np from xarray.testing import _assert_internal_invariants

ds = xr.Dataset() ds.coords["1"] = ("1", np.array([1], dtype=np.uint32)) ds["1_"] = ("1", np.array([1], dtype=np.uint32)) ds = ds.rename_vars({"1": "0"}) ds ```

It looks like this sequence of operations creates a default index

But then ```python from xarray.testing import _assert_internal_invariants

_assert_internal_invariants(ds, check_default_indexes=True) fails with ... File ~/repos/xarray/xarray/testing/assertions.py:301, in _assert_indexes_invariants_checks(indexes, possible_coord_variables, dims, check_default) 299 if check_default: 300 defaults = default_indexes(possible_coord_variables, dims) --> 301 assert indexes.keys() == defaults.keys(), (set(indexes), set(defaults)) 302 assert all(v.equals(defaults[k]) for k, v in indexes.items()), ( 303 indexes, 304 defaults, 305 )

AssertionError: ({'0'}, set()) ```

What is your issue?

when dask-expr is fixed. Added in https://github.com/pydata/xarray/pull/8837

Not for merging, just proof that it can be done nicely :)

Now builds on #8840 ~Builds on an older version of #8507~

Try it out!

```python import xarray as xr from xarray.core.groupers import SeasonGrouper, SeasonResampler

ds = xr.tutorial.open_dataset("air_temperature")

custom seasons!

ds.air.groupby(time=SeasonGrouper(["JF", "MAM", "JJAS", "OND"])).mean()

ds.air.resample(time=SeasonResampler(["DJF", "MAM", "JJAS", "ON"])).count() ```

upstream-dev env is broken

• [x] Closes #8623

• [x] Closes #8830
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

What happened?

All tests are failing because of an error in create_test_data

from xarray.tests import create_test_data create_test_data()

```

AssertionError Traceback (most recent call last) Cell In[3], line 2 1 from xarray.tests import create_test_data ----> 2 create_test_data()

File ~/repos/xarray/xarray/tests/init.py:329, in create_test_data(seed, add_attrs, dim_sizes) 327 obj.coords["numbers"] = ("dim3", numbers_values) 328 obj.encoding = {"foo": "bar"} --> 329 assert all(var.values.flags.writeable for var in obj.variables.values()) 330 return obj

AssertionError: ```

xref https://github.com/pydata/xarray/actions/runs/8269168819/job/22623800305?pr=8777

Is your feature request related to a problem?

If we have two objects with dims x and x1, then xr.align(..., join="exact") will pass because these dimensions are broadcastable.

I'd like a stricter option (join="strict"?) that disallows broadcasting.

Describe the solution you'd like

python xr.align( xr.DataArray([1], dims="x"), xr.DataArray([1], dims="x1"), join="strict", ) would raise an error.

It'd be nice to have this as a built-in option so we can use python with xr.set_options(arithmetic_join="strict"): ...

Describe alternatives you've considered

An alternative would be to allow control over automatic broadcasting through the set_options context manager., but that seems like it would be more complicated to implement.

Additional context

This turns up in staggered grid calculations with xgcm where it is easy to mistakenly construct very high-dimensional arrays because of automatic broadcasting.

Some refactoring towards the Grouper refactor described in #8510

1. Rename to Resampler from ResampleGrouper
2. Refactor to a single "ResolvedGrouper" object that encapsulates the underling Grouper/Resampler object: UniqueGrouper, BinGrouper, or TimeResampler.

xref #8509, #6610

Rendered version here

@pydata/xarray I've been poking at this on and off for a year now and finally figured out how to do it cleanly (#8509).

I wrote up a design doc for Grouper objects that allow custom conversion of DataArrays to integer group codes, following the NEP template (which is absolutely great!). Such Grouper objects allow us to generalize the GroupBy interface to a much larger class of problems, and eventually provide a nice path to grouping by multiple variables (#6610)

8509 implements two custom Groupers for you to try out :)

```python import xarray as xr from xarray.core.groupers import SeasonGrouper, SeasonResampler

ds = xr.tutorial.open_dataset("air_temperature")

custom seasons!

ds.air.groupby(time=SeasonGrouper(["JF", "MAM", "JJAS", "OND"])).mean()

ds.air.resample(time=SeasonResampler(["DJF", "MAM", "JJAS", "ON"])).count() ```

All comments are welcome, 1. there are a couple of specific API and design decisions to be made. I'll make some comments pointing these out. 2. I'm also curious about what Grouper objects we should provide in Xarray.

cc @ilan-gold @ivirshup @aulemahal @tomvothecoder @jbusecke @katiedagon - it would be good to hear what "Groupers" would be useful for your work / projects. I bet you already have examples that fit this proposal

Is your feature request related to a problem?

The defaults for concat are excessively permissive: data_vars="all", coords="different", compat="no_conflicts", join="outer". This comment illustrates why this can be hard to predict or understand: a seemingly unrelated option decode_cf controls whether a variable is in data_vars or coords, and can result in wildly different concatenation behaviour.

1. This always concatenates data_vars along concat_dim even if they did not have that dimension to begin with.
2. If the same coordinate var exists in different datasets/files, they will be sequentially compared for equality to decide whether they get concatenated.
3. The outer join (applied along all dimensions that are not concat_dim) can result in very large datasets due to small floating points differences in the indexes, and also questionable behaviour with staggered grid datasets.
4. "no_conflicts" basically picks the first not-NaN value after aligning all datasets, but is quite slow (we should be using duck_array_ops.nanfirst here I think).

While "convenient" this really just makes the default experience quite bad with hard-to-understand slowdowns.

Describe the solution you'd like

I propose we migrate to data_vars="minimal", coords="minimal", join="exact", compat="override". This should 1. only concatenate data_vars and coords variables when they already have concat_dim. 2. For any variables that do not have concat_dim, it will blindly pick them from the first file. 3. join="exact" will prevent ballooning of dimension sizes due to floating point inequalities. 4. These options will totally avoid any data reads unless explicitly requested by the user.

Unfortunately, this has a pretty big blast radius so we'd need a long deprecation cycle.

Describe alternatives you've considered

No response

Additional context

xref https://github.com/pydata/xarray/issues/4824 xref https://github.com/pydata/xarray/issues/1385 xref https://github.com/pydata/xarray/issues/8231 xref https://github.com/pydata/xarray/issues/5381 xref https://github.com/pydata/xarray/issues/2064 xref https://github.com/pydata/xarray/issues/2217

What is your issue?

Thanks to @keewis for volunteering at today's meeting :()

This fixes the failure locally for me.

cc @max-sixty

• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

What is your issue?

We haven't done this yet! https://github.com/pydata/xarray/blob/d87ba61c957fc3af77251ca6db0f6bccca1acb82/pyproject.toml#L11-L15

What is your issue?

Thanks @TomNicholas for volunteering to drive this release!

I haven't looked at this too closely but it appears that this is a way to save MultiIndexed datasets to netCDF. So we may be able to do sparse -> multiindex -> netCDF

http://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#compression-by-gathering

cc @fujiisoup

• [x] xref #2157, xref #1460
• [x] Closes #8518, closes #8263
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

🤦🏾‍♂️

Spotted in https://github.com/pydata/xarray/pull/8460

Builds on #8586

What is your issue?

Looks like we can bump a number of min versions: ``` Package Required Policy Status

cartopy 0.20 (2021-09-17) 0.21 (2022-09-10) < dask-core 2022.7 (2022-07-08) 2022.12 (2022-12-02) < distributed 2022.7 (2022-07-08) 2022.12 (2022-12-02) < flox 0.5 (2022-05-03) 0.6 (2022-10-12) < iris 3.2 (2022-02-15) 3.4 (2022-12-01) < matplotlib-base 3.5 (2021-11-18) 3.6 (2022-09-16) < numba 0.55 (2022-01-14) 0.56 (2022-09-28) < numpy 1.22 (2022-01-03) 1.23 (2022-06-23) < packaging 21.3 (2021-11-18) 22.0 (2022-12-08) < pandas 1.4 (2022-01-22) 1.5 (2022-09-19) < scipy 1.8 (2022-02-06) 1.9 (2022-07-30) < seaborn 0.11 (2020-09-08) 0.12 (2022-09-06) < typing_extensions 4.3 (2022-07-01) 4.4 (2022-10-07) < zarr 2.12 (2022-06-23) 2.13 (2022-09-27) < ```

• [x] Closes #8581
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Using 255 as fillvalue for int8 arrays will not be allowed any more. Previously this overflowed to -1. Now specify that instead.

On numpy 1.24.4 ```

np.array([255], dtype="i1") DeprecationWarning: NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays. The conversion of 255 to int8 will fail in the future.

array([-1], dtype=int8) ```

Spotted when profiling some write workloads. 1. Speeds up the check for multi-index 2. Speeds up one string encoder by not re-creating variables when not necessary.

@benbovy is there a better way?

What is your issue?

https://docs.xarray.dev/en/stable/getting-started-guide/installing.html#minimum-dependency-versions says that we follow NEP-29, and I think our min versions script also does that.

I propose we follow https://scientific-python.org/specs/spec-0000/

In practice, I think this means we mostly drop Python versions earlier.

Builds on #8560

• [x] Closes #8409
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

cc @max-sixty

``` print(len(cloudpickle.dumps(da.chunk(lat=1, lon=1).map_blocks(lambda x: x))))

779354739 -> 47699827

print(len(cloudpickle.dumps(da.chunk(lat=1, lon=1).drop_vars(da.indexes).map_blocks(lambda x: x))))

15981508

```

This is a quick attempt. I think we can generalize this to minimize duplication.

The downside is that the graphs are not totally embarrassingly parallel any more. This PR:

vs main:

Discussed in https://github.com/pydata/xarray/discussions/8567

Fixes roundtripping of string dtype indexes

• [x] Tests added

The new asv version comes with decorators for parameterizing and skipping, and the ability to use mamba to create environments.

https://github.com/airspeed-velocity/asv/releases

https://asv.readthedocs.io/en/v0.6.1/writing_benchmarks.html#skipping-benchmarks

This might help us reduce benchmark times a bit, or at least simplify the code some.

Is your feature request related to a problem?

When parallel=True and a distributed client is active, Xarray reads every file in parallel, constructs a Dataset per file with indexed coordinates loaded, and then sends all of that back to the "head node" for the combine.

Instead we can tree-reduce the combine (example) by switching to dask.bag instead of dask.delayed and skip the overhead of shipping 1000s of copies of an indexed coordinate back to the head node.

1. The downside is the dask graph is "worse" but perhaps that shouldn't stop us.
2. I think this is only feasible for combine="nested"

cc @TomNicholas

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] Closes #2157
• [ ] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Could use a close-ish review.

What happened?

xarray/namedarray/core.py:758: error: Value of type Never is not indexable [index] xarray/core/alignment.py:684: error: Unused "type: ignore" comment [unused-ignore] xarray/core/alignment.py:1156: error: Unused "type: ignore" comment [unused-ignore] xarray/core/dataset.py: note: In member "sortby" of class "Dataset": xarray/core/dataset.py:7967: error: Incompatible types in assignment (expression has type "tuple[Alignable, ...]", variable has type "tuple[DataArray, ...]") [assignment] xarray/core/dataset.py:7979: error: "Alignable" has no attribute "isel" [attr-defined]

We already get the underlying Zarr array in https://github.com/pydata/xarray/blob/bb8511e0894020e180d95d2edb29ed4036ac6447/xarray/backends/zarr.py#L529-L531

and then pass it to open_store_variable. Just pass that array down to ZarrArrayWrapper instead of reading it from the datastore again.

Avoid repeatedly pinging a remote store by requesting keys at one go.

Is your feature request related to a problem?

We should document changing the time vector using pandas date offsets here

This is particularly useful for centering the time stamps after a resampling operation.

Related: - CFTime offsets: https://github.com/pydata/xarray/issues/5687 - loffset deprecation: https://github.com/pydata/xarray/pull/7444

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

xref #8448 get back to green checks for now.

Also fixes the RTD failure on main

• [x] Closes https://github.com/pydata/xarray/issues/7971
• [x] Closes https://github.com/pydata/xarray/issues/8437.

We were previously pinned to <0.21 Removing the pin didn't change the env but with >=0.21 we get 0.22 which works.

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Should update the date and delete empty sections before merging

• [x] fix breaking attrs test
• [x] Look at numeric_only in NAMED_ARRAY_OBJECT
• xref #8238
• [ ] Tests added

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

This gives us better reprs where only differing attributes are shown in the diff.

On main: ... Differing coordinates: L * x (x) <U1 'a' 'b' foo: bar same: same R * x (x) <U1 'a' 'c' source: 0 foo: baz same: same ...

With this PR: ``` Differing coordinates: L * x (x) %cU1 'a' 'b' Differing variable attributes: foo: bar R * x (x) %cU1 'a' 'c' Differing variable attributes: source: 0 foo: baz

```

Use as_indexable instead of NumpyIndexingAdapter

What is your issue?

Raising @Illviljan's comment from https://github.com/pydata/xarray/pull/8304#discussion_r1363196597.

• [x] Closes #7764
• [x] Closes #8017
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Is your feature request related to a problem?

Shall we support opt_einsum as an optional backend for xr.dot?

opt_einsum.contract is a drop-in replacement for np.einsum so this monkey-patch works today xr.core.duck_array_ops.einsum = opt_einsum.contract

Describe the solution you'd like

Add a backend kwarg with options "numpy" and "opt_einsum", with the default being "numpy"

Describe alternatives you've considered

We could create a new package but it seems a bit silly.

Additional context

No response

Is your feature request related to a problem?

We can do plain reductions, and groupby reductions with numbagg. Rolling is the last one left!

I don't think coarsen will benefit since it's basically a reshape and reduce on that view, so it should already be accelerated. There may be small gains in handling the boundary conditions but that's probably it.

What happened?

Here's a simple rescaling calculation: ```python import numpy as np import xarray as xr

ds = xr.Dataset( {"a": (("x", "y"), np.ones((300, 400))), "b": (("x", "y"), np.ones((300, 400)))} ) mean = ds.mean() # scalar std = ds.std() # scalar rescaled = (ds - mean) / std ```

The profile for the last line shows 30% (!!!) time spent in align (really reindex_like) except there's nothing to reindex when only scalars are involved!

This is a small example inspired by a ML pipeline where this normalization is happening very many times in a tight loop.

cc @benbovy

What did you expect to happen?

A fast path for when no reindexing needs to happen.

xref #8339 xref #8350

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [ ] Tests added - check bottleneck tests
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

What is your issue?

As we work through refactoring for NamedArray, it's pretty clear that Xarray will depend pretty closely on many files in namedarray/. For example various utils.py, pycompat.py, *ops.py, formatting.py, formatting_html.py at least. This promises to be quite painful if we did break NamedArray out in to its own repo (particularly around typing, e.g. https://github.com/pydata/xarray/pull/8309)

I propose we use pydata/xarray as a monorepo that serves two packages: NamedArray and Xarray. - We can move as much as is needed to have NamedArray be independent of Xarray, but Xarray will depend quite closely on many utility functions in NamedArray. - We can release both at the same time similar to dask and distributed. - We can re-evaluate if and when NamedArray grows its own community.

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

What is your issue?

Copying over @max-sixty's comment from https://github.com/pydata/xarray/pull/8304#discussion_r1358873522

I haven't been in touch with the NameArray discussions so forgive a glib comment — but re https://github.com/pydata/xarray/issues/3891 — this would be a "once-in-a-library" opportunity to always retain attrs in aggregations, removing the keep_attrs option in methods.

(Xarray could still handle them as it wished, so xarray's external interface wouldn't need to change immediately...)

@pydata/xarray Should we just delete the keep_attrs kwarg completely for NamedArray and always propagate attrs? obj.attrs.clear() seems just as easy to type.

What is your issue?

Here's the output from the Scientific Python Repo Review tool.

There's an online version here.

On mac I run pipx run 'sp-repo-review[cli]' --format html --show err gh:pydata/xarray@main | pbcopy

A lot of these seem fairly easy to fix. I'll note that there's a large number of mypy config suggestions.

General

• Detected build backend: setuptools.build_meta
• Detected license(s): Apache Software License

Projects must have a noxfile.py or tox.ini to encourage new contributors.

PyProject

See https://github.com/pydata/xarray/issues/8239#issuecomment-1739363809

xfail_strict should be set. You can manually specify if a check should be strict when setting each xfail.

[tool.pytest.ini_options]
xfail_strict = true

-ra should be in addopts = [...] (print summary of all fails/errors).

[tool.pytest.ini_options]
addops = ["-ra", "--strict-config", "--strict-markers"]

Pre-commit

Use https://github.com/psf/black-pre-commit-mirror instead of https://github.com/psf/black in .pre-commit-config.yaml

Must have https://github.com/codespell-project/codespell repo in .pre-commit-config.yaml

Must have https://github.com/pre-commit/pygrep-hooks repo in .pre-commit-config.yaml

Must have https://github.com/pre-commit/mirrors-prettier repo in .pre-commit-config.yaml

If --fix is present, --show-fixes must be too.

Should have something like this in .pre-commit-config.yaml:

ci:
  autoupdate_commit_msg: 'chore: update pre-commit hooks'

MyPy

Must have strict in the mypy config. MyPy is best with strict or nearly strict configuration. If you are happy with the strictness of your settings already, ignore this check or set strict = false explicitly.

[tool.mypy]
strict = true

Must have warn_unreachable = true to pass this check. There are occasionally false positives (often due to platform or Python version static checks), so it's okay to ignore this check. But try it first - it can catch real bugs too.

[tool.mypy]
warn_unreachable = true

Must have "ignore-without-code" in enable_error_code = [...]. This will force all skips in your project to include the error code, which makes them more readable, and avoids skipping something unintended.

[tool.mypy]
enable_error_code = ["ignore-without-code", "redundant-expr", "truthy-bool"]

Must have "redundant-expr" in enable_error_code = [...]. This helps catch useless lines of code, like checking the same condition twice.

[tool.mypy]
enable_error_code = ["ignore-without-code", "redundant-expr", "truthy-bool"]

Must have "truthy-bool" in enable_error_code = []. This catches mistakes in using a value as truthy if it cannot be falsey.

[tool.mypy]
enable_error_code = ["ignore-without-code", "redundant-expr", "truthy-bool"]

Ruff

Must select the flake8-bugbear B checks. Recommended:

[tool.ruff]
select = [
  "B",  # flake8-bugbear
]