1. This moves the region detection code into ZarrStore so we only open the store once.
2. Instead of opening the store as a dataset, construct a pd.Index directly to "auto"-infer the region.

The diff is large mostly because a bunch of code moved from backends/api.py to backends/zarr.py

• [x] Closes #5733
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Like pandas.date_range, cftime_range would previously return dates outside the range of the specified start and end dates if provided a negative frequency: ```

start = cftime.DatetimeGregorian(2023, 10, 31) end = cftime.DatetimeGregorian(2021, 11, 1) xr.cftime_range(start, end, freq="-1YE") CFTimeIndex([2023-12-31 00:00:00, 2022-12-31 00:00:00, 2021-12-31 00:00:00], dtype='object', length=3, calendar='standard', freq='-1YE-DEC') ```

This PR ports a bug fix from pandas (https://github.com/pandas-dev/pandas/issues/56147) to prevent this from happening. The above example now produces: ```

start = cftime.DatetimeGregorian(2023, 10, 31) end = cftime.DatetimeGregorian(2021, 11, 1) xr.cftime_range(start, end, freq="-1YE") CFTimeIndex([2022-12-31 00:00:00, 2021-12-31 00:00:00], dtype='object', length=2, calendar='standard', freq=None) ```

Since this is a bug fix, we do not make any attempt to preserve the old behavior if an earlier version of pandas is installed. In the testing context this means we skip some tests for pandas versions less than 3.0.

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] towards #8844

This "fixes" the upstream-dev failures related to the removal of numpy.array_api. There are a couple of open questions, though: - array-api-strict is not installed by default, so namedarray would get a new dependency. Not sure how to deal with that – as far as I can tell, numpy.array_api was not supposed to be used that way, so maybe we need to use array-api-compat instead? What do you think, @andersy005, @Illviljan? - array-api-strict does not define Array.nbytes (causing a funny exception that wrongly claims DataArray does not define nbytes) - array-api-strict has a different DType class, which makes it tricky to work with both numpy dtypes and said dtype class in the same code. In particular, if I understand correctly we're supposed to check dtypes using isdtype, but numpy.isdtype will only exist in numpy>=2, array-api-strict's version does not define datetime / string / object dtypes, and numpy.issubdtype does not work with the non-numpy dtype class). So maybe we need to use array-api-compat internally?

Builds on #8997

This removes the use of the deprecated numpy.core._exceptions.UFuncError (and multiplication as a way to attach units), and makes sure we run the pint tests in the upstream-dev CI again.

This pull request is the first part of migrating the indexing and broadcasting logic from xarray.core.variable to xarray.namedarray. I intend to open follow-up pull requests to address additional changes related to this refactoring, as outlined in the proposal for decoupling lazy indexing functionality from NamedArray.

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] towards #8844

In the tests, we've been calling np.cross with vectors of 2 or 3 dimensions, numpy>=2 will deprecate 2D vectors (plus, we're now raising on warnings). Thus, we 0-pad the inputs before generating the expected result (which generally should not change the outcome of the tests).

For a later PR: add tests to check if xr.cross works if more than a single dimension is present, and pre-compute the expected result. Also, for property-based testing: the cross-product of two vectors is perpendicular to both input vectors (use the dot product to check that), and its length (l2-norm) is the product of the lengths of the input vectors.

• [ ] Towards #8856
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Per https://github.com/pydata/xarray/issues/8844#issuecomment-2089427222, for the time being this marks test_use_cftime_false_standard_calendar_in_range as an expected failure under NumPy 2. Hopefully we'll be able to fix the upstream issue in pandas eventually.

I considered wedging this into core/ops.py, but it didn't look like it fit there.

This is a basic lift and shift from datatree_/ops.py to core/datatree_ops.py

I did fix the document addendum injection and added a couple of tests.

• [x] Contributes to migration step for miscellaneous modules in Track merging datatree into xarray #8572
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

If we create a Coordinates object using the concatenated result_indexes, and pass that to the Dataset constructor, we can explicitly set the correct indexes from the start, instead of auto-creating the wrong ones and then trying to overwrite them with the correct indexes later (which is what the current implementation does).

• [x] Possible fix for #8871
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

- [x] towards #8844

• replace numpy.core.finfo with numpy.finfo
• add dtype and copy parameters to all definitions of __array__

• [x] Completes deprecation cycle described in #8496, and started in #8500
• [ ] ~~Tests added~~
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

In trying to install packages that are compatible with numpy>=2 I added several projects that are built in CI without build isolation (so that they will be built with the nightly version of numpy). That was a temporary workaround, so we should start thinking about cleaning this up.

As it seems numcodecs is now compatible (or uses less of numpy in compiled code, not sure), this is an attempt to see if CI works if we use the version from conda-forge.

bottleneck and cftime now build against numpy>=2.0.0rc1, so we can stop building them without build isolation.

Don't just xfail, and not only on windows, since it can crash the worker

The example in the doc-string of the Dataset class prior to this commit uses an example array whose size is 2 x 2 x 3 with the first two dimensions labeled "x" and "y" and the final dimension labeled "time". This was confusing due to the fact that "x" and "y" are just arbitrary names for these axes and that no reason is given for the data to be organized in a 2x2x3 array instead of a 2x2 matrix. This commit clarifies the example.

Additionally, this PR contains updates to the documentation, specifically the user-guide/data-structures.rst file; the updates bring the documentation examples into alignment with the doc-string change. Unfortunately, I wasn't able to build the documentation, so this will need to be checked. (I followed the instructions here, but despite cfgrib working fine, I got an error about how it wasn't a valid engine.)

See issue #8970 for more information.

• [X] Closes #8970

this PR builds towards

• https://github.com/pydata/xarray/pull/8870
• https://github.com/pydata/xarray/pull/8856

the primary objective is to partially address

1. Implement fall back .oindex, .vindex properties on BackendArray base class. These will simply rewrap the key tuple with the appropriate *Indexer object, and pass it on to __getitem__ or __setitem__. These methods will also raise DeprecationWarning so that external backends will know to migrate to .oindex, and .vindex over the next year.

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

As noted in https://github.com/pydata/xarray/issues/8909, the use of .oindex and .vindex properties in coding/* appears to have broken some backends (e.g. scipy). This PR reverts those changes. We plan to bundle these changes into a separate backends feature branch (see this comment, which will be merged once we are confident about its impact on downstream dependencies.

• [ ] Closes #8909
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Notes: * For duckarrays, use _dtype[_generic] * For actual np.ndarrays, use np.dtype[np.generic] * np.dtype is too specific in general and it's probably not needed in most of their array_api functions. * _DTypeBase-class in np? * Mixing dtypes from np and xp is discouraged: https://github.com/data-apis/array-api/issues/582 * Using asarray seems to be the recommended way.

• [x] Starts the deprecation cycle for #8959
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~
• [ ] Change existing code + examples so as not to emit this new warning everywhere.

I think this is the final set of methods

• [x] Closes #6646
• [ ] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Notes inline. Would fix #4587

This brings in the option to stack all dimensions except for one or more dimensions listed. I find this very useful for quickly iterating over all the combinations of dimensions except for one (i.e. you have a number of input parameters that are parameterized and one time dimension, and you want to calculate some time response for all the combinations of these input parameters and store them in the time-row corresponding to the appropriate combination of inputs).

I played around with implementing to_stacked_array() for DataArray, but this made less sense in the end since that method was really designed for Datasets.

• [x] Addresses #8278
• [ ] Tests added (added one for DataArray, just now realizing I should have one for Dataset)
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

1. Now that numba supports 3.12.
2. Disabled pint on the main environment since it doesn't work. Pint is still installed in the all-but-dask env, which is still runs python 3.11 for this reason.

Turns out pydap also needs to build with numpy>=2. Until it does, we should remove it from the upstream-dev environment. Also, numcodecs build-depends on setuptools-scm.

And finally, the h5py nightlies might support numpy>=2 (h5py>=3.11 supposedly is numpy>=2 compatible), so once again I'll try and see if CI passes.

• [x] towards #8844

See #8402 for more discussion. Bottom line is that numpy 2 changes the rules for casting between two inputs. Due to this and xarray's preference for promoting python scalars to 0d arrays (scalar arrays), xarray objects are being upcast to higher data types when they previously didn't.

I'm mainly opening this PR for further and more detailed discussion.

CC @dcherian

• [ ] Closes #8402
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Notes are inline.

• [x] Closes https://github.com/pydata/xarray/issues/8494
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

Done with some help from an LLM — quite good for doing tedious tasks that we otherwise wouldn't want to do — can paste in all the warnings output and get a decent start on rules for exclusions

While I don't feel super confident that this is designed to protect against any bugs, it does solve the immediate problem in #8371, by hoisting the encoding check above the code that runs for only new variables. The encoding check is somewhat implicit, so this was an easy thing to miss prior.

• [x] Closes #8371,
• [x] Closes #8882
• [x] Closes #8876
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] Closes first step of #8496, would require another PR later to actually change the return type. Also really resolves the second half of #921.
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

• [x] Solves part of #8871 by pulling out part of https://github.com/pydata/xarray/pull/8872#issuecomment-2027571714
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

TODO: - [x] Add new kwarg to DataArray.expand_dims - [ ] Add examples to docstrings? - [x] Check it actually solves the problem in #8872

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

This PR should never be merged, it is opened only to run the build-the-docs with the changes from #8976

I just wanted to make sure I could point to what the final doc pages will look like when datatree is released.

This PR continues the overall work of migrating DataTree into xarray.

• xarray/core/datatree_render.py is the renamed version of xarray/datatree_/datatree/render.py.
• xarray/core/extensions.py now contains functionality from xarray/datatree_/datatree/extensions.py.
• xarray/core/formatting.py now contains functionality from xarray/datatree_/datatree/formatting.py.
• xarray/tests/test_datatree.py now contains tests from xarray/datatree_/datatree/tests/test_dataset_api.py.
• xarray/testing/assertions.py now contains functionality from /xarray/datatree_/datatree/testing.py.

I had also meant to get to common.py and what's left of io.py, but I've got a hefty couple of days of meetings ahead, so I wanted to get this progress into PR before that happens. @flamingbear or I can follow up with the remaining things in a separate PR. (Also this PR is already getting a little big, so maybe it's already got enough in it)

• [x] Contributes to migration step for miscellaneous modules in #8572
• [ ] ~~Tests added~~
• [ ] ~~User visible changes (including notable bug fixes) are documented in whats-new.rst~~
• [ ] ~~New functions/methods are listed in api.rst~~

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

A small follow-on to #7019 to allow using non-dask implementations of delayed.

(Builds off of #7019)

• [x] Closes #7810
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

When opening files from path-like objects (str, pathlib.Path), the backend machinery (_dataset_from_backend_dataset) sets the "source" encoding. This is useful if we need the original path for additional processing, like writing to a similarly named file, or to extract additional metadata. This would be useful as well when using fsspec to open remote files.

In this PR, I'm extracting the path attribute that most fsspec objects have to set that value. I've considered using isinstance checks instead of the getattr-with-default, but the list of potential classes is too big to be practical (at least 4 classes just within fsspec itself).

If this sounds like a good idea, I'll update the documentation of the "source" encoding to mention this feature.

• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

This PR removes the directive that strips out the datatree_ directory from the xarray distribution.

It also cleans a few typing errors and removes exceptions for the datatree_ directory for mypy.

It does NOT remove the exception for pre-commit config.

• [X] Closes #8768

• [X] Closes #7748
• [X] Tests added - test_dataset_diff_dim_nonexist, test_dataarray_diff_dim_nonexist
• [X] User visible changes (including notable bug fixes) are documented in whats-new.rst

FYI @aulemahal, numpy.NaN will be removed in the upcoming numpy=2.0 release.

• [x] follow-up to #8603

Expose Grouper and Resampler as public API

TODO: - [ ] Consider avoiding IndexVariable

• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

• [x] Closes #8665
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

This PR migrates the formatting_html.py module into xarray/core/formatting_html.py as part of the on-going effort to merge xarray-datatree into xarray.

One thing of note is that importing and setting the OPTIONS to "default" in datatree/formatting_html.py (lines) were moved into xarray/core/options.py on #L23 and #L49. So, I did not add them back to xarray/core/formatting_html.py.

• [x] Completes migration step for datatree/formating_htmls.py https://github.com/pydata/xarray/issues/8572
• [x] Tests added
• [ ] ~~User visible changes (including notable bug fixes) are documented in whats-new.rst~~
• [ ] ~~New functions/methods are listed in api.rst~~

Some outstanding points/decisions brought up by this PR: - [ ] Confirm type promotion rules and write them out. As it stands now, if everything is of the same extension array type, it is passed onwards and otherwise is converted to numpy. (related: https://github.com/pydata/xarray/pull/8714) ~- [ ] Acceptance of plum as a dispatch method. Without it, the behavior should be fallen back on from before (cast to numpy types). I am a big fan of dispatching and think it could serve as a model going forward for making support of other data types/arrays more feasible. The other option, I think, would be to just use the underlying array of the ExtensionDuckArray class to decide and then have some central registry that serves as the basis for a decorator (like the api for accessors via _CachedAccessor). That being said, the current defaults are quite good so this is a marginal feature, in all likelihood.~ - [ ] Do we allow just pandas ExtensionArray directly or can we also allow Series?

Possible missing something else! Let me know!

Checklist: - [x] Closes #8463 and Closes #5287 - [x] Tests added - [x] User visible changes (including notable bug fixes) are documented in whats-new.rst - [ ] New functions/methods are listed in api.rst

This PR continues the overall work of migrating DataTree into xarray.

datatree_mapping.py is the renamed version of mapping.py from the datatree repository.

• [x] Closes migration step for mapping.py #8572
• [ ] ~~Tests added~~
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

• [ ] Attempt at fixing #8646
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

I've tried here re-implementing swap_dims using rename_dims, drop_indexes and set_xindex. This fixes the example in #8646 but unfortunately this fails at handling the pandas multi-index special case (i.e., a single non-dimension coordinate wrapping a pd.MultiIndex that is promoted to a dimension coordinate in swap-dims auto-magically results in a PandasMultiIndex with both dimension and level coordinates).

• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Small PR to add a new "method" to convert to and from 360_day calendars. The current two methods (chosen with the align_on keyword) will always remove or add the same day-of-year for all years of the same length.

This new option will randomly chose the days, one for each fifth of the year (72-days period). It emulates the method of the LOCA datasets (see web page and article ). February 29th is always removed/added when the source/target is a leap year.

I copied the implementation from xclim (which I wrote), see code here .

Aims to resolve parts of #8844.

python xarray/tests/test_accessor_str.py::test_case_str: AssertionError: assert dtype('<U26') == dtype('<U30') I'm not sure this is the right location for the fix, at least it fixes those errors. AFAICT this is some issue somewhere inside apply_ufunc where the string dtype size is kept. So this fix removes the size information from the dtype (actually recreating it).

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

Bumps the actions group with 1 update: codecov/codecov-action.

Updates codecov/codecov-action from 4.2.0 to 4.3.0

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

This PR continues the overall work of migrating DataTree into xarray.

iterators.py does not have direct tests. In discussions with @TomNicholas and @flamingbear, we concurred that other unit tests utilise this functionality.

• [x] Closes migration step for iterators.py #8572
• [ ] ~~Tests added~~
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

According to our policy (and NEP-29) we can drop support for python=3.9 since about a week ago. Interestingly, SPEC0 says we could have started doing this about half a year ago (Q4 2023).

We could delay this until we have a release that is compatible with numpy>=2.0, though (numpy>=2.1 will drop support for python=3.9).

• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

Noticed the **kwargs part of the docstring was mangled - see here

• [ ] ~~Closes #xxxx~~
• [ ] ~~Tests added~~
• [ ] ~~User visible changes (including notable bug fixes) are documented in whats-new.rst~~
• [ ] ~~New functions/methods are listed in api.rst~~

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Since cftime version 1.3.0, the base cftime.datetime object can be calendar-aware, obviating the need for calendar-specific subclasses like cftime.DatetimeNoLeap. This PR aims to finally enable the use of these objects in xarray. We can also use this moment to remove cruft around accommodating inexact cftime datetime arithmetic, since that has been fixed since cftime version 1.2.0.

The idea will be to support both for a period of time and eventually drop support for the calendar-specific subclasses. I do not think too much should need to change within xarray—the main challenge will be to see if we can maintain adequate test coverage without multiplying the number of cftime tests by two. This draft PR is at least a start towards that.

• [ ] Closes #4336
• [ ] Closes #4853
• [ ] Closes #5551
• [ ] Closes #8298
• [ ] Closes #8941
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] follow-up to #8846

pandas recently removed the deprecated unit kwarg to TimedeltaIndex.

• [x] towards #8844

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Issue #2921 is about mismatching time units between a time variable and its "bounds" companion. However, #2965 does more than fixing #2921, it removes all double attributes from "bounds" variables which has the undesired side effect that there is currently no way to save them to netcdf with xarray. Since the mentioned link is a recommendation and not a hard requirement for CF compliance, these attributes should be left to the caller to prepare the dataset variables appropriately if required. Reduces the amount of surprise that attributes are not written to disk and fixes #8368.

• [x] Closes #8368
• [ ] Tests added (simple round trip test would be in issue #8368)
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst (not sure about "user visibility" here)

This fix enhances the issue 8860. I did not add any test since I believe it is not needed for this case since we did not add any functionality.

• [X] Closes #8860
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

A drive-by PR as I was trying to figure out how these functions works.

Bumps the actions group with 1 update: codecov/codecov-action.

Updates codecov/codecov-action from 4.1.1 to 4.2.0

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

This adds a line plotter based on LineCollections, called .lines at the moment.

I wanted to replace darray.plot() with using LineCollection instead. But unfortunately due to how many cases are supported (and tested in xarray) darray.plot() will continue using plt.plot.

xref:

4820

5622

Update the cache always.

Enforce typing on all tests in test_plot.py and add the remaining type hints.

@benbovy this seems to be the root cause of #8646, the variable name in Dataset._variables does not match IndexVariable.name.

A good number of tests seem to fail though, so not sure if this is a good chck.

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] Attempt to closes issue described in https://github.com/pydata/xarray/issues/8566#issuecomment-1870472827
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] Closes #8753

This seems to have been around since 2016-ish, so presumably our backend code path is passing arrays around, not Variables.

cc @ilan-gold

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

I was curious to see if the hypothesis stateful testing would catch an inconsistent sequence of index manipulation operations like #8646. Turns out rename_vars is basically broken? (filed #8659) :P

PS: this blog post is amazing. E state = DatasetStateMachine() E state.assert_invariants() E > === E E <xarray.Dataset> E Dimensions: () E Data variables: E *empty* E === E E E > vars: ('1', '1_') E state.add_dim_coord(var=<xarray.Variable (1: 1)> E array([0], dtype=uint32)) E state.assert_invariants() E > === E E <xarray.Dataset> E Dimensions: (1: 1) E Coordinates: E * 1 (1) uint32 0 E Data variables: E 1_ (1) uint32 0 E === E E E > renaming 1 to 0 E state.rename_vars(newname='0') E state.assert_invariants() E > === E E <xarray.Dataset> E Dimensions: (1: 1) E Coordinates: E * 0 (1) uint32 0 E Dimensions without coordinates: 1 E Data variables: E 1_ (1) uint32 0 E === E E E state.teardown()

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

joining the party - [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Closes #8252 Closes #3921

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Toward #8510, builds on #8776

This adds a proposal for decoupling the lazy indexing array machinery, indexing adapter machinery, and Variable's setitem and getitem methods, so that the latter can be migrated to NamedArray.

cc @andersy005

• [x] xref #8844

• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

Small follow up to #8877. If we're going to drop the indices anyways for region writes, we may as well not raise if they are still in the dataset. This makes the user experience of region writes simpler:

```python ds = xr.tutorial.open_dataset("air_temperature") ds.to_zarr("test.zarr") region = {"time": slice(0, 10)}

This fails unless we remember to ds.drop_vars(["lat", "lon"])

ds.isel(**region).to_zarr("test.zarr", region=region) ```

I find this annoying because I often have a dataset with a bunch of unrelated indexes and have to remember which ones to drop, or use some verbose set logic. I thought #8877 might have already done this, but not quite. By just reordering the point at which we drop indices, we can now skip this. We still raise if data vars are passed that don't overlap with the region.

cc @dcherian

_wrap_numpy_scalars relies on np.isscalar, which incorrectly labels a single cftime object as not a scalar.

```python import cftime import numpy as np

c = cftime.datetime(2000, 1, 1, calendar='360_day') np.isscalar(c) # False ```

The PR adds logic to handle non-numpy objects using the np.ndim function. The logic for built-ins and numpy objects should remain the same.

The function logic could possibly be rewritten more clearly as ```python

if hasattr(array, "dtype"):
    if np.isscalar(array):
        return np.array(array)
    else:
        return array

if np.ndim(array) == 0:
    return np.array(array)

return array

```

• [x] Closes #7794
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] Closes https://github.com/xarray-contrib/xeofs/issues/148
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

The change from https://github.com/pydata/xarray/pull/8672 mostly fixed the issue of serializing a reset multiindex in the backends, but there was an additional niche issue that turned up in xeofs that was causing serialization to still fail on the zarr backend.

The issue is that zarr is the only backend that uses a custom version of encode_cf_variable called encode_zarr_variable, and the way this gets called we don't pass through the name of the variable before running ensure_not_multiindex.

As a minimal fix, this PR just passes name through as an additional arg to the general encode_variable function. See @benbovy's comment that maybe we should actually unwrap the level coordinate in reset_index and clean up the checks in ensure_not_multiindex, but I wasn't able to get that working easily.

The exact workflow this turned up in involves DataTree and looks like this: ```python import numpy as np import xarray as xr from datatree import DataTree

ND DataArray that gets stacked along a multiindex

da = xr.DataArray(np.ones((3, 3)), coords={"dim1": [1, 2, 3], "dim2": [4, 5, 6]}) da = da.stack(feature=["dim1", "dim2"])

Extract just the stacked coordinates for saving in a dataset

ds = xr.Dataset(data_vars={"feature": da.feature})

Reset the multiindex, which should make things serializable

ds = ds.reset_index("feature") dt1 = DataTree() dt2 = DataTree(name="feature", data=ds) dt1["foo"] = dt2

Somehow in this step, dt1.foo.feature.dim1.variable becomes an IndexVariable again

print(type(dt1.foo.feature.dim1.variable))

Works

dt1.to_netcdf("test.nc", mode="w")

Fails

dt1.to_zarr("test.zarr", mode="w") ```

But we can reproduce in xarray with the test added here.

• Updates the docstring for persist to mention that it is not altering the original object.

• Adds a return value to the docstring for compute and persist on both Dataset and DataArray

• [x] Closes #8901

• [x] Closes #8343
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

The intended pattern is: ```python

after_init = initialize_zarr(store, ds, region_dims=("x",))
for i in range(ds.sizes["x"]):
    after_init.isel(x=[i]).to_zarr(store, region={"x": slice(i, i + 1)})

```

cc @slevang

updates: - github.com/astral-sh/ruff-pre-commit: v0.2.0 → v0.3.4 - github.com/psf/black-pre-commit-mirror: 24.1.1 → 24.3.0 - github.com/pre-commit/mirrors-mypy: v1.8.0 → v1.9.0

Bumps the actions group with 1 update: codecov/codecov-action.

Updates codecov/codecov-action from 4.1.0 to 4.1.1

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Should have been done as part of the last release https://github.com/pydata/xarray/releases/tag/v2024.03.0

Adds a whole suite of hypothesis strategies for generating xarray objects, inspired by and separated out from the new hypothesis strategies in #4972. They are placed into the namespace xarray.testing.strategies, and publicly mentioned in the API docs, but with a big warning message. There is also a new testing page in the user guide documenting how to use these strategies.

• [x] Closes #6911
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

EDIT: A variables strategy and user-facing documentation were shipped in https://github.com/pydata/xarray/pull/8404

Enforce typing on all tests in test_groupby.py and add the remaining type hints.

• [x] Closes #8887, and probably supersedes #8809
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• ~~New functions/methods are listed in api.rst~~

@slevang this should also make work your test case added in #8809. I haven't added it here, instead I added a basic check that should be enough.

I don't really understand why the serialization backends (zarr?) do not seem to work with the PandasMultiIndexingAdapter.__array__() implementation, which should normally coerce the multi-index levels into numpy arrays as needed. Anyway, I guess that coercing it early like in this PR doesn't hurt and may avoid the confusion of a non-indexed, isolated coordinate variable that still wraps a pandas.MultiIndex.

Upstream numpy has become a bit more particular with which types you can use for inplace operations. This PR fixes ``` __ TestImshow.test_imshow_rgb_values_in_valid_range __

self = <xarray.tests.test_plot.TestImshow object at 0x7f88320c2780>

def test_imshow_rgb_values_in_valid_range(self) -> None:
    da = DataArray(np.arange(75, dtype="uint8").reshape((5, 5, 3)))
    _, ax = plt.subplots()

  out = da.plot.imshow(ax=ax).get_array()

/home/runner/work/xarray/xarray/xarray/tests/test_plot.py:2034:

/home/runner/work/xarray/xarray/xarray/plot/accessor.py:421: in imshow return dataarray_plot.imshow(self._da, args, kwargs) /home/runner/work/xarray/xarray/xarray/plot/dataarray_plot.py:1601: in newplotfunc primitive = plotfunc( /home/runner/work/xarray/xarray/xarray/plot/dataarray_plot.py:1853: in imshow alpha = 255

self = masked_array( data=[[[1], [1], [1], [1], [1]],

    [[1],
     [1],
...,
     [1],
     [1],
     [1],
     [1]]],

mask=False, fill_value=np.int64(999999), dtype=uint8) other = 255

def __imul__(self, other):
    """
    Multiply self by other in-place.

    """
    m = getmask(other)
    if self._mask is nomask:
        if m is not nomask and m.any():
            self._mask = make_mask_none(self.shape, self.dtype)
            self._mask += m
    elif m is not nomask:
        self._mask += m
    other_data = getdata(other)
    other_data = np.where(self._mask, other_data.dtype.type(1), other_data)

  self._data.__imul__(other_data)

E numpy._core._exceptions._UFuncOutputCastingError: Cannot cast ufunc 'multiply' output from dtype('int64') to dtype('uint8') with casting rule 'same_kind'

/home/runner/micromamba/envs/xarray-tests/lib/python3.12/site-packages/numpy/ma/core.py:4415: UFuncTypeError ```

Some curious behaviors seen while debugging: ```python alpha = np.array([1], dtype=np.int8) alpha *= 255 repr(alpha) # 'array([-1], dtype=int8)'

alpha = np.array([1], dtype=np.int16) alpha *= 255 repr(alpha) # 'array([255], dtype=int16)' ```

xref: #8844

Simply adds datetime.date to plottable types in _ensure_plottable in plot/utils.pyL675 to enable the plotting of dates.

Matplotlib handles date types automatically, so I think there is no other change needed.

Do I need to add a test for this? Any pointers on where I would put it that?

• [x] Closes #8866
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

We need to read existing variables to make sure we append or write to a region with the right encoding. Currently we decode all arrays in a Zarr group. Instead only decode those arrays for which we require encoding information.

• [x] Solves the problem in https://github.com/pydata/xarray/pull/8712#issuecomment-1929037299
• [ ] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Supercedes #8884 as a way to close #8883, in light of me having learnt that this is now allowed! https://github.com/pydata/xarray/issues/8883#issuecomment-2024645815. So this is really a follow-up to #7989.

• [x] Closes #8883
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~