1. Now that numba supports 3.12.
2. Disabled pint on the main environment since it doesn't work. Pint is still installed in the all-but-dask env, which is still runs python 3.11 for this reason.

What is your issue?

cc @hmaarrfk @kmuehlbauer

Passed: https://github.com/pydata/xarray/actions/runs/4115923717/jobs/7105298426 Failed: https://github.com/pydata/xarray/actions/runs/4115946392/jobs/7105345290

Versions: h5netcdf 1.1.0 pyhd8ed1ab_0 conda-forge h5py 3.8.0 nompi_py310h5555e59_100 conda-forge hdf4 4.2.15 h7aa5921_5 conda-forge hdf5 1.12.2 nompi_h48135f9_101 conda-forge

``` =================================== FAILURES =================================== ___ test_open_mfdataset_manyfiles[h5netcdf-20-True-5-5] ______ [gw1] darwin -- Python 3.10.9 /Users/runner/micromamba-root/envs/xarray-tests/bin/python

readengine = 'h5netcdf', nfiles = 20, parallel = True, chunks = 5 file_cache_maxsize = 5

@requires_dask
@pytest.mark.filterwarnings("ignore:use make_scale(name) instead")
def test_open_mfdataset_manyfiles(
    readengine, nfiles, parallel, chunks, file_cache_maxsize
):
    # skip certain combinations
    skip_if_not_engine(readengine)

    if ON_WINDOWS:
        pytest.skip("Skipping on Windows")

    randdata = np.random.randn(nfiles)
    original = Dataset({"foo": ("x", randdata)})
    # test standard open_mfdataset approach with too many files
    with create_tmp_files(nfiles) as tmpfiles:
        writeengine = readengine if readengine != "pynio" else "netcdf4"
        # split into multiple sets of temp files
        for ii in original.x.values:
            subds = original.isel(x=slice(ii, ii + 1))
            if writeengine != "zarr":
                subds.to_netcdf(tmpfiles[ii], engine=writeengine)
            else:  # if writeengine == "zarr":
                subds.to_zarr(store=tmpfiles[ii])

        # check that calculation on opened datasets works properly

      with open_mfdataset(
            tmpfiles,
            combine="nested",
            concat_dim="x",
            engine=readengine,
            parallel=parallel,
            chunks=chunks if (not chunks and readengine != "zarr") else "auto",
        ) as actual:

/Users/runner/work/xarray/xarray/xarray/tests/test_backends.py:3267:

/Users/runner/work/xarray/xarray/xarray/backends/api.py:991: in open_mfdataset datasets, closers = dask.compute(datasets, closers) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/base.py:599: in compute results = schedule(dsk, keys, kwargs) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/threaded.py:89: in get results = get_async( /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py:511: in get_async raise_exception(exc, tb) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py:319: in reraise raise exc /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/local.py:224: in execute_task result = _execute_task(task, data) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/core.py:119: in _execute_task return func((_execute_task(a, cache) for a in args)) /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/dask/utils.py:72: in apply return func(args, kwargs) /Users/runner/work/xarray/xarray/xarray/backends/api.py:526: in open_dataset backend_ds = backend.open_dataset( /Users/runner/work/xarray/xarray/xarray/backends/h5netcdf_.py:417: in open_dataset ds = store_entrypoint.open_dataset( /Users/runner/work/xarray/xarray/xarray/backends/store.py:32: in open_dataset vars, attrs = store.load() /Users/runner/work/xarray/xarray/xarray/backends/common.py:129: in load (decode_variable_name(k), v) for k, v in self.get_variables().items() /Users/runner/work/xarray/xarray/xarray/backends/h5netcdf.py:220: in get_variables return FrozenDict( /Users/runner/work/xarray/xarray/xarray/core/utils.py:471: in FrozenDict return Frozen(dict(args, *kwargs)) /Users/runner/work/xarray/xarray/xarray/backends/h5netcdf_.py:221: in <genexpr> (k, self.open_store_variable(k, v)) for k, v in self.ds.variables.items() /Users/runner/work/xarray/xarray/xarray/backends/h5netcdf_.py:200: in open_store_variable elif var.compression is not None: /Users/runner/micromamba-root/envs/xarray-tests/lib/python3.10/site-packages/h5netcdf/core.py:394: in compression return self._h5ds.compression

self = <[AttributeError("'NoneType' object has no attribute '_root'") raised in repr()] Variable object at 0x151378970>

@property
def _h5ds(self):
    # Always refer to the root file and store not h5py object
    # subclasses:

  return self._root._h5file[self._h5path]

E AttributeError: 'NoneType' object has no attribute '_h5file'

```

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Update the cache always.

Discussed in https://github.com/pydata/xarray/discussions/8751

• [x] Closes #8753

This seems to have been around since 2016-ish, so presumably our backend code path is passing arrays around, not Variables.

cc @ilan-gold

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

I was curious to see if the hypothesis stateful testing would catch an inconsistent sequence of index manipulation operations like #8646. Turns out rename_vars is basically broken? (filed #8659) :P

PS: this blog post is amazing. E state = DatasetStateMachine() E state.assert_invariants() E > === E E <xarray.Dataset> E Dimensions: () E Data variables: E *empty* E === E E E > vars: ('1', '1_') E state.add_dim_coord(var=<xarray.Variable (1: 1)> E array([0], dtype=uint32)) E state.assert_invariants() E > === E E <xarray.Dataset> E Dimensions: (1: 1) E Coordinates: E * 1 (1) uint32 0 E Data variables: E 1_ (1) uint32 0 E === E E E > renaming 1 to 0 E state.rename_vars(newname='0') E state.assert_invariants() E > === E E <xarray.Dataset> E Dimensions: (1: 1) E Coordinates: E * 0 (1) uint32 0 E Dimensions without coordinates: 1 E Data variables: E 1_ (1) uint32 0 E === E E E state.teardown()

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

joining the party - [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Closes #8252 Closes #3921

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Toward #8510, builds on #8776

This adds a proposal for decoupling the lazy indexing array machinery, indexing adapter machinery, and Variable's setitem and getitem methods, so that the latter can be migrated to NamedArray.

cc @andersy005

• [x] xref #8844

We need to read existing variables to make sure we append or write to a region with the right encoding. Currently we decode all arrays in a Zarr group. Instead only decode those arrays for which we require encoding information.

What happened?

Currently we don't overwrite indexes when region="auto" https://github.com/pydata/xarray/blob/e6ccedb56ed4bc8d0b7c1f16ab325795330fb19a/xarray/backends/api.py#L1769-L1770

I propose we do this for all region writes and completely disallow modifying indexes with a region write.

This would match the map_blocks model, where all indexes are specified in the template and no changes by the mapped function are allowed.

• [x] Closes #8589
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

cc @slevang

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

@aulemahal would you be able to test against xclim's test suite. I imagine you're doing a bunch of grouped quantiles.

…roadcast=False`` (#8784)"

This reverts commit 11f89ecdd41226cf93da8d1e720d2710849cd23e.

Reverting #8784

Sadly that PR broke a lot of tests by breaking create_test_data with from xarray.tests import create_test_data create_test_data()

```

AssertionError Traceback (most recent call last) Cell In[3], line 2 1 from xarray.tests import create_test_data ----> 2 create_test_data()

File ~/repos/xarray/xarray/tests/init.py:329, in create_test_data(seed, add_attrs, dim_sizes) 327 obj.coords["numbers"] = ("dim3", numbers_values) 328 obj.encoding = {"foo": "bar"} --> 329 assert all(var.values.flags.writeable for var in obj.variables.values()) 330 return obj

AssertionError: ```

Somehow that code changes whether IndexVariable.values returns a writeable numpy array. I spent some time debugging but couldn't figure it out.

cc @etienneschalk

xref #8844 Closes https://github.com/xarray-contrib/flox/issues/342

• [x] Closes #8843
• [x] Tests added

```python import numpy as np import pandas as pd

pd.set_option("mode.copy_on_write", True)

from xarray.core.variable import _possibly_convert_objects

string_var = np.array(["a", "bc", "def"], dtype=object) datetime_var = np.array( ["2019-01-01", "2019-01-02", "2019-01-03"], dtype="datetime64[ns]" ) assert _possibly_convert_objects(string_var).flags.writeable assert _possibly_convert_objects(datetime_var).flags.writeable ```

The core issue is that we now get read-only arrays back from pandas here: https://github.com/pydata/xarray/blob/fbcac7611bf9a16750678f93483d3dbe0e261a0a/xarray/core/variable.py#L197-L212

@phofl is this expected?

What is your issue?

This line fails with pd.set_options("mode.copy_on_write", True) https://github.com/pydata/xarray/blob/c9d3084e98d38a7a9488380789a8d0acfde3256f/xarray/tests/init.py#L329

We'll need to fix this before Pandas 3 is released in April: https://github.com/pydata/xarray/blob/c9d3084e98d38a7a9488380789a8d0acfde3256f/xarray/tests/init.py#L329

Here's a test ```python def example(): obj = Dataset() obj["dim2"] = ("dim2", 0.5 * np.arange(9)) obj["time"] = ("time", pd.date_range("2000-01-01", periods=20) print({k: v.data.flags for k, v in obj.variables.items()}) return obj

example() pd.set_options("mode.copy_on_write", True) example() ```

What happened?

(xref #8658)

I'm not sure what the expected behaviour is here:

```python import xarray as xr import numpy as np from xarray.testing import _assert_internal_invariants

ds = xr.Dataset() ds.coords["1"] = ("1", np.array([1], dtype=np.uint32)) ds["1_"] = ("1", np.array([1], dtype=np.uint32)) ds = ds.rename_vars({"1": "0"}) ds ```

It looks like this sequence of operations creates a default index

But then ```python from xarray.testing import _assert_internal_invariants

_assert_internal_invariants(ds, check_default_indexes=True) fails with ... File ~/repos/xarray/xarray/testing/assertions.py:301, in _assert_indexes_invariants_checks(indexes, possible_coord_variables, dims, check_default) 299 if check_default: 300 defaults = default_indexes(possible_coord_variables, dims) --> 301 assert indexes.keys() == defaults.keys(), (set(indexes), set(defaults)) 302 assert all(v.equals(defaults[k]) for k, v in indexes.items()), ( 303 indexes, 304 defaults, 305 )

AssertionError: ({'0'}, set()) ```

upstream-dev env is broken

• [x] Closes #8623

• [x] Closes #8830
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

What happened?

All tests are failing because of an error in create_test_data

from xarray.tests import create_test_data create_test_data()

```

AssertionError Traceback (most recent call last) Cell In[3], line 2 1 from xarray.tests import create_test_data ----> 2 create_test_data()

File ~/repos/xarray/xarray/tests/init.py:329, in create_test_data(seed, add_attrs, dim_sizes) 327 obj.coords["numbers"] = ("dim3", numbers_values) 328 obj.encoding = {"foo": "bar"} --> 329 assert all(var.values.flags.writeable for var in obj.variables.values()) 330 return obj

AssertionError: ```

xref https://github.com/pydata/xarray/actions/runs/8269168819/job/22623800305?pr=8777

Is your feature request related to a problem?

If we have two objects with dims x and x1, then xr.align(..., join="exact") will pass because these dimensions are broadcastable.

I'd like a stricter option (join="strict"?) that disallows broadcasting.

Describe the solution you'd like

python xr.align( xr.DataArray([1], dims="x"), xr.DataArray([1], dims="x1"), join="strict", ) would raise an error.

It'd be nice to have this as a built-in option so we can use python with xr.set_options(arithmetic_join="strict"): ...

Describe alternatives you've considered

An alternative would be to allow control over automatic broadcasting through the set_options context manager., but that seems like it would be more complicated to implement.

Additional context

This turns up in staggered grid calculations with xgcm where it is easy to mistakenly construct very high-dimensional arrays because of automatic broadcasting.

Some refactoring towards the Grouper refactor described in #8510

1. Rename to Resampler from ResampleGrouper
2. Refactor to a single "ResolvedGrouper" object that encapsulates the underling Grouper/Resampler object: UniqueGrouper, BinGrouper, or TimeResampler.

xref #8509, #6610

Rendered version here

@pydata/xarray I've been poking at this on and off for a year now and finally figured out how to do it cleanly (#8509).

I wrote up a design doc for Grouper objects that allow custom conversion of DataArrays to integer group codes, following the NEP template (which is absolutely great!). Such Grouper objects allow us to generalize the GroupBy interface to a much larger class of problems, and eventually provide a nice path to grouping by multiple variables (#6610)

8509 implements two custom Groupers for you to try out :)

```python import xarray as xr from xarray.core.groupers import SeasonGrouper, SeasonResampler

ds = xr.tutorial.open_dataset("air_temperature")

custom seasons!

ds.air.groupby(time=SeasonGrouper(["JF", "MAM", "JJAS", "OND"])).mean()

ds.air.resample(time=SeasonResampler(["DJF", "MAM", "JJAS", "ON"])).count() ```

All comments are welcome, 1. there are a couple of specific API and design decisions to be made. I'll make some comments pointing these out. 2. I'm also curious about what Grouper objects we should provide in Xarray.

cc @ilan-gold @ivirshup @aulemahal @tomvothecoder @jbusecke @katiedagon - it would be good to hear what "Groupers" would be useful for your work / projects. I bet you already have examples that fit this proposal

What is your issue?

Thanks to @keewis for volunteering at today's meeting :()

This fixes the failure locally for me.

cc @max-sixty

• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

What is your issue?

We haven't done this yet! https://github.com/pydata/xarray/blob/d87ba61c957fc3af77251ca6db0f6bccca1acb82/pyproject.toml#L11-L15

What is your issue?

Thanks @TomNicholas for volunteering to drive this release!

• [x] xref #2157, xref #1460
• [x] Closes #8518, closes #8263
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

🤦🏾‍♂️

Spotted in https://github.com/pydata/xarray/pull/8460

Builds on #8586

What is your issue?

Looks like we can bump a number of min versions: ``` Package Required Policy Status

cartopy 0.20 (2021-09-17) 0.21 (2022-09-10) < dask-core 2022.7 (2022-07-08) 2022.12 (2022-12-02) < distributed 2022.7 (2022-07-08) 2022.12 (2022-12-02) < flox 0.5 (2022-05-03) 0.6 (2022-10-12) < iris 3.2 (2022-02-15) 3.4 (2022-12-01) < matplotlib-base 3.5 (2021-11-18) 3.6 (2022-09-16) < numba 0.55 (2022-01-14) 0.56 (2022-09-28) < numpy 1.22 (2022-01-03) 1.23 (2022-06-23) < packaging 21.3 (2021-11-18) 22.0 (2022-12-08) < pandas 1.4 (2022-01-22) 1.5 (2022-09-19) < scipy 1.8 (2022-02-06) 1.9 (2022-07-30) < seaborn 0.11 (2020-09-08) 0.12 (2022-09-06) < typing_extensions 4.3 (2022-07-01) 4.4 (2022-10-07) < zarr 2.12 (2022-06-23) 2.13 (2022-09-27) < ```

• [x] Closes #8581
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Using 255 as fillvalue for int8 arrays will not be allowed any more. Previously this overflowed to -1. Now specify that instead.

On numpy 1.24.4 ```

np.array([255], dtype="i1") DeprecationWarning: NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays. The conversion of 255 to int8 will fail in the future.

array([-1], dtype=int8) ```

Spotted when profiling some write workloads. 1. Speeds up the check for multi-index 2. Speeds up one string encoder by not re-creating variables when not necessary.

@benbovy is there a better way?

Builds on #8560

• [x] Closes #8409
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

cc @max-sixty

``` print(len(cloudpickle.dumps(da.chunk(lat=1, lon=1).map_blocks(lambda x: x))))

779354739 -> 47699827

print(len(cloudpickle.dumps(da.chunk(lat=1, lon=1).drop_vars(da.indexes).map_blocks(lambda x: x))))

15981508

```

This is a quick attempt. I think we can generalize this to minimize duplication.

The downside is that the graphs are not totally embarrassingly parallel any more. This PR:

vs main:

Fixes roundtripping of string dtype indexes

• [x] Tests added

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] Closes #2157
• [ ] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Could use a close-ish review.

What happened?

xarray/namedarray/core.py:758: error: Value of type Never is not indexable [index] xarray/core/alignment.py:684: error: Unused "type: ignore" comment [unused-ignore] xarray/core/alignment.py:1156: error: Unused "type: ignore" comment [unused-ignore] xarray/core/dataset.py: note: In member "sortby" of class "Dataset": xarray/core/dataset.py:7967: error: Incompatible types in assignment (expression has type "tuple[Alignable, ...]", variable has type "tuple[DataArray, ...]") [assignment] xarray/core/dataset.py:7979: error: "Alignable" has no attribute "isel" [attr-defined]

We already get the underlying Zarr array in https://github.com/pydata/xarray/blob/bb8511e0894020e180d95d2edb29ed4036ac6447/xarray/backends/zarr.py#L529-L531

and then pass it to open_store_variable. Just pass that array down to ZarrArrayWrapper instead of reading it from the datastore again.

Avoid repeatedly pinging a remote store by requesting keys at one go.

Is your feature request related to a problem?

We should document changing the time vector using pandas date offsets here

This is particularly useful for centering the time stamps after a resampling operation.

Related: - CFTime offsets: https://github.com/pydata/xarray/issues/5687 - loffset deprecation: https://github.com/pydata/xarray/pull/7444

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

xref #8448 get back to green checks for now.

Also fixes the RTD failure on main

• [x] Closes https://github.com/pydata/xarray/issues/7971
• [x] Closes https://github.com/pydata/xarray/issues/8437.

We were previously pinned to <0.21 Removing the pin didn't change the env but with >=0.21 we get 0.22 which works.

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Should update the date and delete empty sections before merging

• [x] fix breaking attrs test
• [x] Look at numeric_only in NAMED_ARRAY_OBJECT
• xref #8238
• [ ] Tests added

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

This gives us better reprs where only differing attributes are shown in the diff.

On main: ... Differing coordinates: L * x (x) <U1 'a' 'b' foo: bar same: same R * x (x) <U1 'a' 'c' source: 0 foo: baz same: same ...

With this PR: ``` Differing coordinates: L * x (x) %cU1 'a' 'b' Differing variable attributes: foo: bar R * x (x) %cU1 'a' 'c' Differing variable attributes: source: 0 foo: baz

```

• [x] Closes #7764
• [x] Closes #8017
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Is your feature request related to a problem?

Shall we support opt_einsum as an optional backend for xr.dot?

opt_einsum.contract is a drop-in replacement for np.einsum so this monkey-patch works today xr.core.duck_array_ops.einsum = opt_einsum.contract

Describe the solution you'd like

Add a backend kwarg with options "numpy" and "opt_einsum", with the default being "numpy"

Describe alternatives you've considered

We could create a new package but it seems a bit silly.

Additional context

No response

xref #8339 xref #8350

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [ ] Tests added - check bottleneck tests
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

We've accumulated a nice number of changes.

Can someone volunteer to do a release in the next few days?

• [x] Closes #8106
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

cc @blaylockbk

Is your feature request related to a problem?

Consider this example

python ds = xr.Dataset({"a": ("x", [1, 2, 3]), "b": ("y", [1, 2, 3])}) xr.apply_ufunc(np.mean, ds, input_core_dims=[["x"]])

This raises ValueError: operand to apply_ufunc has required core dimensions ['x'], but some of these dimensions are absent on an input variable: ['x']

because core dimension x is missing on variable b. This behaviour makes it annoying to use apply_ufunc on Datasets.

Describe the solution you'd like

Add a new kwarg to apply_ufunc called missing_core_dim that controls how to handle variables without all input core dimensions. This kwarg could take one of two values: 1. "raise" - raise an error, current behaviour 2. "copy" - skip applying the function and copy the variable from input to output. 3. "drop"- skip applying the function and drop the variable.

Describe alternatives you've considered

No response

Additional context

No response

What happened?

See https://github.com/pydata/xarray/pull/8101

FAILED xarray/tests/test_missing.py::test_interpolate_pd_compat - ValueError: 'fill_value' is not a valid keyword for DataFrame.interpolate FAILED xarray/tests/test_missing.py::test_interpolate_pd_compat_non_uniform_index - ValueError: 'fill_value' is not a valid keyword for DataFrame.interpolate

and this doctest FAILED xarray/core/dataarray.py::xarray.core.dataarray.DataArray.to_unstacked_dataset @pydata/xarray can someone take a look please?

Should say that Y and M cannot be supported with datetime64

Discussed in https://github.com/pydata/xarray/discussions/8000

Is your feature request related to a problem?

I opened this PR to add minimum versions for our optional dependencies: https://github.com/conda-forge/xarray-feedstock/pull/84/files to prevent issues like #7467

I think we'd need a policy to choose which ones to list. Here's the current list: run_constrained: - bottleneck >=1.3 - cartopy >=0.20 - cftime >=1.5 - dask-core >=2022.1 - distributed >=2022.1 - flox >=0.5 - h5netcdf >=0.13 - h5py >=3.6 - hdf5 >=1.12 - iris >=3.1 - matplotlib-base >=3.5 - nc-time-axis >=1.4 - netcdf4 >=1.5.7 - numba >=0.55 - pint >=0.18 - scipy >=1.7 - seaborn >=0.11 - sparse >=0.13 - toolz >=0.11 - zarr >=2.10

Some examples to think about: 1. iris seems like a bad one to force. It seems like people might use Iris and Xarray independently and Xarray shouldn't force a minimum version. 2. For backends, I arbitrarily kept netcdf4, h5netcdf and zarr. 3. It seems like we should keep array types: so dask, sparse, pint.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

We could even go down to quarterly

• [x] Closes #7683
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

What happened?

From https://discourse.pangeo.io/t/xarray-unable-to-allocate-memory-how-to-size-up-problem/3233/4

Consider ``` python

ds is dataset with big dask arrays

mean = ds.groupby("time.day").mean() mean.to_netcdf() mean = xr.open_dataset(...)

ds.groupby("time.day") - mean ```

In GroupBy._binary_op https://github.com/pydata/xarray/blob/39caafae4452f5327a7cd671b18d4bb3eb3785ba/xarray/core/groupby.py#L616

we will eagerly construct other that is of the same size as ds.

What did you expect to happen?

I think the only solution is to automatically chunk if ds has dask arrays, and other (or mean) isn't backed by dask arrays. A chunk size of 1 seems sensible.

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

What happened?

I just ran in to this error in an environment without dask. TypeError: Could not find a Chunk Manager which recognises type <class 'dask.array.core.Array'>

I think we could easily recommend the user to install a package that provides dask by looking at type(array).__name__. This would make the message a lot friendlier

• [x] Closes #7766
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

@gsieros can you try this out please?

• [x] Closes #2233
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Avoid automatic creating of Index variable when nD variable shares name with one of its dimensions.

Closes #2233

```python url = "http://www.smast.umassd.edu:8080/thredds/dodsC/FVCOM/NECOFS/Forecasts/NECOFS_GOM3_FORECAST.nc" ds = xr.open_dataset(url, engine="netcdf4") display(ds)

xr.testing._assert_internal_invariants(ds, check_default_indexes=False) ! no raise on #7368 ```

~The internal invariants assert fails on main but succeeds on #7368~. EDIT: now fixed the invariants check.

What is your issue?

See https://github.com/matplotlib/matplotlib/issues/16296

Looks like the MPL API may have advanced enough that we can delete some of our use of private attributes.

• [x] Closes #7841
• [x] Closes #7892

Builds on the refactoring in #7206

• [x] xref #6610
• [x] Use TimeResampleGrouper

• [x] Closes #6836
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] Closes #7199
• [x] Tests ~added~ deleted
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

xref #7730

What happened?

```python with xr.set_options(display_style="text", use_flox=False): with xr.set_options(use_flox=False): display( xr.DataArray( data=np.array([np.nan, 1, 1, np.nan, 1, 1]), dims="x", coords={"labels": ("x", np.array([1, 2, 3, 1, 2, 3]))}, ) .groupby("labels") .sum() )

with xr.set_options(use_flox=True):
    display(
        xr.DataArray(
            data=np.array([np.nan, 1, 1, np.nan, 1, 1]),
            dims="x",
            coords={"labels": ("x", np.array([1, 2, 3, 1, 2, 3]))},
        )
        .groupby("labels")
        .sum()
    )

```

```

without flox

<xarray.DataArray (labels: 3)> array([0., 2., 2.]) Coordinates: * labels (labels) int64 1 2 3

with flox

<xarray.DataArray (labels: 3)> array([nan, 2., 2.]) Coordinates: * labels (labels) int64 1 2 3 ```

What did you expect to happen?

The same answer. We should set min_count=0 when min_count is None

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

• [x] Closes #7808
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] Closes #7797
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

cc @slevang

Pandas is expecting to release v2 in two weeks (pandas-dev/pandas#46776 (comment)). But we are still incompatible with their main branch: - #7441 - #7420

This PR pins pandas to <2