If we create a Coordinates object using the concatenated result_indexes, and pass that to the Dataset constructor, we can explicitly set the correct indexes from the start, instead of auto-creating the wrong ones and then trying to overwrite them with the correct indexes later (which is what the current implementation does).

• [x] Possible fix for #8871
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

• [x] Completes deprecation cycle described in #8496, and started in #8500
• [ ] ~~Tests added~~
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

• [x] Starts the deprecation cycle for #8959
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~
• [ ] Change existing code + examples so as not to emit this new warning everywhere.

• [x] Closes first step of #8496, would require another PR later to actually change the return type. Also really resolves the second half of #921.
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

• [x] Solves part of #8871 by pulling out part of https://github.com/pydata/xarray/pull/8872#issuecomment-2027571714
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

TODO: - [x] Add new kwarg to DataArray.expand_dims - [ ] Add examples to docstrings? - [x] Check it actually solves the problem in #8872

A small follow-on to #7019 to allow using non-dask implementations of delayed.

(Builds off of #7019)

• [x] Closes #7810
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] Closes #8665
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Noticed the **kwargs part of the docstring was mangled - see here

• [ ] ~~Closes #xxxx~~
• [ ] ~~Tests added~~
• [ ] ~~User visible changes (including notable bug fixes) are documented in whats-new.rst~~
• [ ] ~~New functions/methods are listed in api.rst~~

• [x] Attempt to closes issue described in https://github.com/pydata/xarray/issues/8566#issuecomment-1870472827
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Should have been done as part of the last release https://github.com/pydata/xarray/releases/tag/v2024.03.0

Adds a whole suite of hypothesis strategies for generating xarray objects, inspired by and separated out from the new hypothesis strategies in #4972. They are placed into the namespace xarray.testing.strategies, and publicly mentioned in the API docs, but with a big warning message. There is also a new testing page in the user guide documenting how to use these strategies.

• [x] Closes #6911
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

EDIT: A variables strategy and user-facing documentation were shipped in https://github.com/pydata/xarray/pull/8404

• [x] Solves the problem in https://github.com/pydata/xarray/pull/8712#issuecomment-1929037299
• [ ] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Supercedes #8884 as a way to close #8883, in light of me having learnt that this is now allowed! https://github.com/pydata/xarray/issues/8883#issuecomment-2024645815. So this is really a follow-up to #7989.

• [x] Closes #8883
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

• [x] Closes #8883
• [x] Tests added
• [ ] ~~User visible changes (including notable bug fixes) are documented in whats-new.rst~~
• [ ] ~~New functions/methods are listed in api.rst~~

Tries fixing #8704 by cherry-picking from #8124 as @benbovy suggested in https://github.com/pydata/xarray/issues/8704#issuecomment-1926868422

• [x] Closes #8704
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

@dcherian I got to this point before realizing that simply moving parallelcompat.py over isn't what it says in the design doc, which instead talks about

• Could this functionality be left in Xarray proper for now? Alternative array types like JAX also have some notion of "chunks" for parallel arrays, but the details differ in a number of ways from the Dask/Cubed.
• Perhaps variable.chunk/load methods should become functions defined in xarray that convert Variable objects. This is easy so long as xarray can reach in and replace .data

I personally think that simply moving parallelcompat makes sense so long as you expect people to use chunked NamedArray objects. I see the chunked arrays as special cases of duck arrays, and my understanding is that NamedArray is supposed to have full support for duckarrays.

cc @andersy005

• [x] As requested in #8238
• [ ] ~~Tests added~~
• [ ] ~~User visible changes (including notable bug fixes) are documented in whats-new.rst~~
• [ ] ~~New functions/methods are listed in api.rst~~

This makes sure we only use the CopyOnWriteArray wrapper on arrays that have been explicitly marked to be lazily-loaded (through being subclasses of BackendArray). Without this change we are implicitly assuming that any array type obtained through the BackendEntrypoint system should be treated as if it points to an on-disk array.

Motivated by https://github.com/pydata/xarray/issues/8699, which is a counterexample to that assumption.

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] Closes #8701
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

How might we want to move headers/sections around to take advantage of now having a navigation bar at the top? Adding an explicit link to the tutorial.xarray.dev site would be good.

• [x] Closes #8666
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Clarify step 8. by pointing to where the ReadTheDocs build actually is

• [x] Hopefully closes #8619

Someone give this a thumbs up if it looks good

• [x] Closes #8616

Builds on top of #4972 to add tests for Variable/DataArray/Dataset constructors and properties when wrapping duck arrays.

Adds a file xarray/tests/duckarrays/base/constructors.py which contains new test base classes.

Also uses those new base classes to test Sparse array integration (not yet tried for pint integration).

• [x] Closes part of #6894
• [ ] Tests added (tests for tests?? Maybe...)
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

I want datatree to live in xarray main, as right now it's in a separate package but imports many xarray internals.

This presents a few questions: 1) At what stage is datatree "ready" to moved in here? At what stage should it become encouraged public API? 2) What's a good way to slowly roll the feature out? 3) How do I decrease the bus factor on datatree's code? Can I get some code reviews during the merging process? :pray: 4) Should I make a new CI environment just for testing datatree stuff?

Today @jhamman and @keewis suggested for now I make it so that you can from xarray import DataTree, using the current xarray-datatree package as an optional dependency. That way I can create a smoother on-ramp, get some more users testing it, but without committing all the code into this repo yet.

@pydata/xarray what do you think? Any other thoughts about best practices when moving a good few thousand lines of code into xarray?

• [x] First step towards moving solution of #4118 into this repository
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

• [x] Needed for https://github.com/tomwhite/cubed/issues/277#issuecomment-1648567431 - should have been added in #7019
• [ ] ~~Tests added~~ (would go in cubed-xarray)
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst (new ABC method will be documented on chunked array types page automatically)

This PR is part of an experiment to see how to integrate a DataTree into xarray.

What is does is refactor Dataset to store variables in a DataManifest class, which is also capable of maintaining a ledger of child tree nodes. The point of this is to prevent name collisions between stored variables and child datatree nodes, as first mentioned in https://github.com/TomNicholas/datatree/issues/38 and explained further in https://github.com/TomNicholas/datatree/issues/2.

("Manifest" in the old sense, of a noun meaning "a document giving comprehensive details of a ship and its cargo and other contents")

• [x] Would eventually close https://github.com/TomNicholas/datatree/issues/38
• [ ] Tests added
• [x] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

In #5961 and #6083 I've been experimenting extending Dataset to store variables in a custom mapping object (instead of always in a dict), so as to eventually fix this mutability problem with DataTree.

I've been writing out new storage class implementations in those PRs, but on Friday @shoyer suggested that I could instead simply alter the allowed type for ._variables in xarray.Dataset's type hints. That would allow me to mess about with storage class implementations outside of xarray, whilst guaranteeing type compatibility with xarray main itself with absolutely minimal changes (hopefully no runtime changes to Dataset at all!).

The idea is to define a protocol in xarray which specifies the structural subtyping behaviour of any custom variable storage class that I might want to set as Dataset._variables. The type hint for the ._variables attribute then refers to this protocol, and will be satisfied as long as whatever object is set as ._variables has compatibly-typed methods. Adding type hints to the ._construct_direct and ._replace constructors is enough to propagate this new type specification all over the codebase.

In practice this means writing a protocol which describes the type behaviour of all the methods on dict that currently get used by ._variable accesses.

So far I've written out a CopyableMutableMapping protocol which defines all the methods needed. The issues I'm stuck on at the moment are:

1) The typing behaviour of overloaded methods, specifically update. (setdefault also has similar problems but I think I can safely omit that from the protocol definition because we don't call ._variables.setdefault() anywhere.) Mypy complains that CopyableMutableMapping is not a compatible type when Dict is specified because the type specification of overloaded methods isn't quite right somehow:

```
xarray/core/computation.py:410: error: Argument 1 to "_construct_direct" of "Dataset" has incompatible type "Dict[Hashable, Variable]"; expected "CopyableMutableMapping[Hashable, Variable]"  [arg-type]
xarray/core/computation.py:410: note: Following member(s) of "Dict[Hashable, Variable]" have conflicts:
xarray/core/computation.py:410: note:     Expected:
xarray/core/computation.py:410: note:         @overload
xarray/core/computation.py:410: note:         def update(self, other: Mapping[Hashable, Variable], **kwargs: Variable) -> None
xarray/core/computation.py:410: note:         @overload
xarray/core/computation.py:410: note:         def update(self, other: Iterable[Tuple[Hashable, Variable]], **kwargs: Variable) -> None
xarray/core/computation.py:410: note:         <1 more overload not shown>
xarray/core/computation.py:410: note:     Got:
xarray/core/computation.py:410: note:         @overload
xarray/core/computation.py:410: note:         def update(self, Mapping[Hashable, Variable], **kwargs: Variable) -> None
xarray/core/computation.py:410: note:         @overload
xarray/core/computation.py:410: note:         def update(self, Iterable[Tuple[Hashable, Variable]], **kwargs: Variable) -> None
```
I don't understand what the inconsistency is because I literally looked up the exact way that [the type stubs](https://github.com/python/typeshed/blob/e6911530d4d52db0fbdf05be3aff89e520ee39bc/stdlib/typing.pyi#L490) for `Dict` were written (via `MutableMapping`).

2) Making functions which expect a Mapping accept my CopyableMutableMapping. I would have thought this would just work because I think my protocol defines all the methods which Mapping has, so CopyableMutableMapping should automatically become a subtype of Mapping. But instead I get errors like this with no further information as to what to do about it.

```xarray/core/dataset.py:785: error: Argument 1 to "Frozen" has incompatible type "CopyableMutableMapping[Hashable, Variable]"; expected "Mapping[Hashable, Variable]"  [arg-type]```

3) I'm expecting to get a runtime problem whenever we assert isinstance(ds._variables, dict), which happens in a few places. I'm no sure what the best way to deal with that is, but I'm hoping that simply adding @typing.runtime_checkable to the protocol class definition will be enough?

Once that passes mypy I will write a test that checks that if I define my own custom variable storage class I can _construct_direct a Dataset which uses it without any errors. At that point I can be confident that Dataset is general enough to hold whichever exact variable storage class I end up needing for DataTree.

@max-sixty this is entirely a typing challenge, so I'm tagging you in case you're interested :)

• [ ] Would supercede #5961 and #6083
• [ ] Tests added
• [ ] Passes pre-commit run --all-files

EDIT: Also using Protocol at all is only available in Python 3.8+

RIP

• [x] Closes #8524

Breaks out just the part of #6908 needed for generating arbitrary xarray.Variable objects. (so ignore the ginormous number of commits)

EDIT: Check out this test which performs a mean on any subset of any Variable object!

```python In [36]: from xarray.testing.strategies import variables

In [37]: variables().example() <xarray.Variable (ĭ: 3)> array([-2.22507386e-313-6.62447795e+016j, nan-6.46207519e+185j, -2.22507386e-309+3.33333333e-001j]) ```

@andersy005 @maxrjones @jhamman I thought this might be useful for the NamedArray testing. (xref #8370 and #8244)

@keewis and @Zac-HD sorry for letting that PR languish for literally a year :sweat_smile: This PR addresses your feedback about accepting a callable that returns a strategy generating arrays. That suggestion makes some things a bit more complex in user code but actually allows me to simplify the internals of the variables strategy significantly. I'm actually really happy with this PR - I think it solves what we were discussing, and is a sensible checkpoint to merge before going back to making strategies for generating composite objects like DataArrays/Datasets work.

• [x] Closes part of #6911
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

• [x] Closes #2226 and #1499 by forbidding those situations (but we should leave #3731 open as the "official" place to discuss supporting repeated dimensions
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Builds upon #7991 by adding a page to the internals enumerating all the different ways in which xarray is interoperable.

Would be nice if https://github.com/pydata/xarray/pull/6975 were merged so that I could link to it from this new page.

• [x] Addresses comment in https://github.com/pydata/xarray/pull/6975#issuecomment-1246487152
• [ ] ~~Tests added~~
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Adds a new .chunksizes property to Dataset, DataArray and Variable, which returns a mapping from dimensions names to chunk sizes in all cases.

Supercedes #5846 because this PR is backwards-compatible.

• [x] Closes #5843
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

• [x] Closes #7962
• [ ] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Documents a hidden feature I noticed yesterday, corrects incorrect docstrings, and tidies up some of the typing internally.

• [ ] ~~Closes #xxxx~~
• [ ] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Another attempt like #5961

@shoyer

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Adds a new page to the xarray internals documentation giving an overview of the internal design of xarray.

This should be helpful for xarray contributors and for developers of extensions because nowhere in the docs does it really explain how DataArray and Dataset are constructed from Variable.

• [ ] ~~Closes #xxxx~~
• [ ] ~~Tests added~~
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Initial attempt to get cubed working within xarray, as an alternative to dask.

• [x] Closes #6807, at least for the case of cubed
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst
• [x] Correct type hints

I've added a manager kwarg to the .chunk methods so you can do da.chunk(manager="cubed") to convert to a chunked cubed.CoreArray, with the default still being da.chunk(manager="dask"). (I couldn't think of a better name than "manager", as "backend" and "executor" are already taken.)

~~At the moment it should work except for an import error that I don't understand, see below.~~

Fro cubed to work at all with this PR we would also need: - [x] Cubed to expose the correct array type consistently https://github.com/tomwhite/cubed/issues/123 - [x] A cubed version of apply_gufunc https://github.com/tomwhite/cubed/pull/119 - implemented in https://github.com/tomwhite/cubed/pull/149 :partying_face:

To-dos for me on this PR: - [x] Re-route xarray.apply_ufunc through cubed.apply_gufunc instead of dask's apply_gufunc when appropriate, - [x] Add from_array_kwargs to opening functions, e.g. open_zarr, and open_dataset, - [x] Add from_array_kwargs to creation functions, such as full_like, - [x] Add store_kwargs as a way to propagate cubed-specific kwargs when saving to_zarr.

To complete this project more generally we should also: - [ ] Have cubed.apply_gufunc support multiple output arguments https://github.com/tomwhite/cubed/issues/152 - [x] Have a top-level cubed.unify_chunks to match dask.array.core.unify_chunks - [ ] Write a test suite for wrapping cubed arrays, which would be best done via #6894 - [ ] Generalise xarray.map_blocks to work on cubed arrays, ideally by first rewriting xarray's implementation of map_blocks to use dask.array.map_blocks

cc @tomwhite

Adds a guide for core developers of xarray. Mostly adapted from napari's core dev guide, but with some extra sections and ideas from the pandas maintainance guide.

@pydata/xarray please give your feedback on this! If you prefer to give feedback in a non-public channel for whatever reason then please use the private core team email.

• [ ] ~~Closes #xxxx~~
• [ ] ~~Tests added~~
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Needed because I started the release process earlier this week by writing a whatsnew, that apparently got merged, but the release hasn't been issued since. I'll self-merge this and release now.

Draft improvements to the user guide page on using duck arrays.

Intended as part of the scipy tutorial effort, though I wasn't sure whether to concentrate on content in the main xarray docs or the tutorial repo.

(I wrote this on a train without enough internet to update my conda environment so I will come back and fix anything that doesn't run.)

• [x] Part of https://github.com/xarray-contrib/xarray-tutorial/issues/170
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

cc @dcherian and @keewis

Builds upon #7911

• [x] Documentation for #7019
• [ ] ~~Tests added~~
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Attempt to fix the pre-commit build failure I keep seeing in the CI (e.g. this failure from https://github.com/pydata/xarray/pull/7881)

Generalizes a small check for chunked arrays in groupby so it now allows cubed arrays through to flox rather than just dask arrays. Does not actually mean that flox groupby will work with cubed yet though, see https://github.com/tomwhite/cubed/issues/223 and https://github.com/xarray-contrib/flox/issues/224

• [x] Should have been done in #7019
• [ ] ~~Tests added~~ (The place to test this would be in [cubed-xarray]
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Release summary:

This release adds features to curvefit, improves the performance of concatenation, and fixes various bugs.

For some reason when I try to use git log "$(git tag --sort=v:refname | tail -1).." --format=%aN | sort -u | perl -pe 's/\n/$1, /' to return the list of all contributors since last release, it only returns Deepak :laughing: I'm not sure what's going wrong there - I definitely have all the git tags fetched, and other people have definitely contributed since the last version!

Follows on from #7067 and #6804, ensuring that we call xp.astype() on arrays rather than arr.astype(), as the latter is commonly-implemented by array libraries but not part of the array API standard.

A bit of a pain to test in isolation because I made the changes so that xarray's .pad would work with array-API-conforming libraries, but actually np.pad is not part of the array API either, so it's going to coerce to numpy for that reason anyway.

(This PR replaces #7815, as making a new branch was easier than merging/rebasing with all the changes in #7019.)

• [ ] ~~Closes #xxxx~~
• [ ] ~~Tests added~~
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Summary:

This release adds some new methods and operators, updates our deprecation policy for python versions, fixes some bugs with groupby, and introduces experimental support for alternative chunked parallel array computation backends via a new plugin system!

While it's common for duck arrays to have a .astype method, this doesn't exist in the new array API standard. We now have duck_array_ops.astype to deal with this, but for some reason changing it in just a couple more places broke practically every pint test in test_units.py :confused: @keewis

Builds on top of #7019 with just one extra commit to separate out this issue.

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Add suggestions @paigem made in #7439, as well as fix a few small formatting things and broken links.

I would like to merge this so that it can be helpful for the new contributors we will hopefully get through Outreachy.

• [x] Closes #7439
• [ ] ~~Tests added~~
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Makes the error message more informative

• [ ] Closes #xxxx
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

• [x] Closes #7533
• [ ] ~~Tests added~~
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

• [x] Closes poor error message reported in https://github.com/pydata/xarray/discussions/7503
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

I think that sphinx-codeautolink is different from sphinx.ext.linkcode...

• [x] Closes #7010
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Change binary operations involving DataTree objects and Dataset objects to be handled by the DataTree class. Necessary to enable ds * dt to return the same type as dt * ds.

Builds on top of #7418.

• [x] Closes https://github.com/xarray-contrib/datatree/issues/146
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

There was a bug in ipython's sphinx extension causing decorators to be skipped when evaluating code blocks. I assume that's why there is this weird workaround in the docs page on defining accessors (which uses decorators).

I fixed that bug, and the fix is in the most recent release of ipython, so this PR bumps our ipython version for the docs, and removes the workaround.

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] Came from https://github.com/pydata/xarray/discussions/7329#discussioncomment-4256845
• [ ] ~~Tests added~~
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

@dcherian

Initial work on integrating the multi-dimensional dask-powered histogram functionality from xhistogram into xarray. Just working on the skeleton to fit around the histogram algorithm for now, to be filled in later.

• [x] Closes #4610
• [x] API skeleton
• [x] Input checking
• [ ] Internal blockwise algorithm from https://github.com/xgcm/xhistogram/pull/49
• [x] Redirect plot.hist
• [x] da.weighted().hist()
• [ ] Tests added for results
• [x] Hypothesis tests for different chunking patterns
• [ ] Examples in documentation
• [ ] Examples in docstrings
• [x] Type hints (first time trying these so might be wrong)
• [ ] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst
• [x] Range argument
• [ ] Handle multidimensional bins (for a future PR? - See https://github.com/xgcm/xhistogram/pull/59)
• [ ] Handle np.datetime64 dtypes by refactoring to use np.searchsorted (for a future PR? See discussion)
• [ ] Fast path for uniform bin widths (for a future PR? See suggestion)

Question: da.hist() or da.histogram()?

There might be a slightly neater way to do this, but this works.

• [x] Closes #5492
• [x] Tests added
• [ ] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] Closes #7232
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

I realised there is very little documentation on Coarsen.construct, so I added this example.

Unsure whether it should instead live in the page on reshaping and reorganising data though, as it is essentially a reshape operation. EDIT: Now on the reshape page

• [ ] ~~Closes #xxxx~~
• [ ] ~~Tests added~~
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

cc @jbusecke @paigem

Thumbs up if it looks fine to you

• [ ] ~~Closes #xxxx~~
• [ ] ~~Tests added~~
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

• [ ] From https://github.com/pydata/xarray/pull/7019#discussion_r968606140
• [ ] ~~Tests added~~
• [ ] ~~User visible changes (including notable bug fixes) are documented in whats-new.rst~~
• [ ] ~~New functions/methods are listed in api.rst~~

As a user, I find not being able to select along one-dimensional non-dimensional coordinates actually comes up fairly often. I think it's quite common to use multiple coordinates to be able to choose between plotting in different coordinate systems (or units) easily.

I've tried to close #2028 in the simplest (but also least efficient) way which was suggested by @shoyer (suggestion 1 here).

This should be temporary anyway: it will get superseded by the explicit indexes refactor. If there is another approach which would achieve the same functionality as this PR but actually bring us closer to #1603 then I would be happy to take a stab at that instead.

I don't really know what to do about the failing test in groupby arithmetic - I think it's caused here but I'm not sure what to replace the triple error type catching (?!) with.

• [x] Closes #2028
• [x] Tests added
• [ ] Passes isort -rc . && black . && mypy . && flake8
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

Adds missing core team members @alexamici and @aurghs to docs, as well as fixing @benbovy 's username.

GH action to automatically label new PRs according to which files they touch.

Idea stolen from dask, see https://github.com/dask/dask/pull/7506 . Their PR labelling by file/module is specified here.

(My first use of this bot so might well be a mistake.)

@max-sixty you will probably enjoy this extra automation :robot:

• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

While making a talk I made a version of our data structure diagram but with added labels along the bottom:

I think this helps clarify the relationship between Variables, DataArrays, and Datasets for new users.

I just made it quickly in inkscape by adding to the previous png - I only realised afterwards that the original was made in LaTeX, so maybe it would be better to add labels directly to that code?

Removes special-casing for dask in duck_array_ops.py, instead relying on NEP-18 to call it when the input is a dask array.

Probably actually don't need the _dask_or_eager_func() (now _module_func()) helper function at all, because all remaining instances look like pandas_isnull = _module_func("isnull", module=pd), which could just be pandas_isnull = pd.isnull.

Only problem is that I seem to have broken one (parameterized) test: test_duck_array_ops.py::test_min_count[True-True-None-sum-True-bool_-1] fails with

```python @pytest.mark.parametrize("dim_num", [1, 2]) @pytest.mark.parametrize("dtype", [float, int, np.float32, np.bool_]) @pytest.mark.parametrize("dask", [False, True]) @pytest.mark.parametrize("func", ["sum", "prod"]) @pytest.mark.parametrize("aggdim", [None, "x"]) @pytest.mark.parametrize("contains_nan", [True, False]) @pytest.mark.parametrize("skipna", [True, False, None]) def test_min_count(dim_num, dtype, dask, func, aggdim, contains_nan, skipna): if dask and not has_dask: pytest.skip("requires dask")

    da = construct_dataarray(dim_num, dtype, contains_nan=contains_nan, dask=dask)
    min_count = 3

    # If using Dask, the function call should be lazy.
    with raise_if_dask_computes():

      actual = getattr(da, func)(dim=aggdim, skipna=skipna, min_count=min_count)

/home/tegn500/Documents/Work/Code/xarray/xarray/tests/test_duck_array_ops.py:578:

/home/tegn500/Documents/Work/Code/xarray/xarray/core/common.py:56: in wrapped_func return self.reduce(func, dim, axis, skipna=skipna, kwargs) /home/tegn500/Documents/Work/Code/xarray/xarray/core/dataarray.py:2638: in reduce var = self.variable.reduce(func, dim, axis, keep_attrs, keepdims, kwargs) /home/tegn500/Documents/Work/Code/xarray/xarray/core/variable.py:1725: in reduce data = func(self.data, kwargs) /home/tegn500/Documents/Work/Code/xarray/xarray/core/duck_array_ops.py:328: in f return func(values, axis=axis, kwargs) /home/tegn500/Documents/Work/Code/xarray/xarray/core/nanops.py:106: in nansum a, mask = _replace_nan(a, 0) /home/tegn500/Documents/Work/Code/xarray/xarray/core/nanops.py:23: in _replace_nan mask = isnull(a) /home/tegn500/Documents/Work/Code/xarray/xarray/core/duck_array_ops.py:83: in isnull return pandas_isnull(data) /home/tegn500/Documents/Work/Code/xarray/xarray/core/duck_array_ops.py:40: in f return getattr(module, name)(args, kwargs) /home/tegn500/miniconda3/envs/py38-mamba/lib/python3.8/site-packages/pandas/core/dtypes/missing.py:127: in isna return _isna(obj) /home/tegn500/miniconda3/envs/py38-mamba/lib/python3.8/site-packages/pandas/core/dtypes/missing.py:166: in _isna return _isna_ndarraylike(np.asarray(obj), inf_as_na=inf_as_na) /home/tegn500/miniconda3/envs/py38-mamba/lib/python3.8/site-packages/numpy/core/_asarray.py:102: in asarray return array(a, dtype, copy=False, order=order) /home/tegn500/miniconda3/envs/py38-mamba/lib/python3.8/site-packages/dask/array/core.py:1502: in array x = self.compute() /home/tegn500/miniconda3/envs/py38-mamba/lib/python3.8/site-packages/dask/base.py:285: in compute (result,) = compute(self, traverse=False, kwargs) /home/tegn500/miniconda3/envs/py38-mamba/lib/python3.8/site-packages/dask/base.py:567: in compute results = schedule(dsk, keys, *kwargs)

self = <xarray.tests.CountingScheduler object at 0x7f0804db2310> dsk = {('xarray-<this-array>-29953318277423606f95b509ad1a9aa7', 0): array([False, False, False, False], dtype=object), ('xar...pe=object), ('xarray-<this-array>-29953318277423606f95b509ad1a9aa7', 3): array([nan, False, False, nan], dtype=object)} keys = [[('xarray-<this-array>-29953318277423606f95b509ad1a9aa7', 0), ('xarray-<this-array>-29953318277423606f95b509ad1a9aa7'...array-<this-array>-29953318277423606f95b509ad1a9aa7', 2), ('xarray-<this-array>-29953318277423606f95b509ad1a9aa7', 3)]] kwargs = {}

def __call__(self, dsk, keys, **kwargs):
    self.total_computes += 1
    if self.total_computes > self.max_computes:

      raise RuntimeError(
            "Too many computes. Total: %d > max: %d."
            % (self.total_computes, self.max_computes)
        )

E RuntimeError: Too many computes. Total: 1 > max: 0.

/home/tegn500/Documents/Work/Code/xarray/xarray/tests/init.py:118: RuntimeError ```

• [x] Closes #5559
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

The spiritual successor to #4696 , this attempts to generalise combine_nested to handle both named and unnamed DataArrays in the same way that combine_by_coords does.

Unfortunately it doesn't actually work yet - I think the problem is a bit more subtle than I originally thought.

Ideally I would implement this using the same logical structure as in #5834, but my attempt to do that was thwarted by how tricky it is to iterate over a nested list-of-lists of arbitrary and modify the stored objects in place...

• [x] Tests added
• [x] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Exposes the inline_array kwarg of dask.array.from_array in xr.open_dataset, and ds/da/variable.chunk.

What setting this to True does is inline the array into the opening/chunking task, which avoids an an extra array object at the start of the task graph. That's useful because the presence of that single common task connecting otherwise independent parts of the graph can confuse the graph optimizer.

With open_dataset(..., inline_array=False):

With open_dataset(..., inline_array=True):

In our case (xGCM) this is important because once inlined the optimizer understands that all the remaining parts of the graph are embarrasingly-parallel, and realizes that it can fuze all our chunk-wise padding tasks into one padding task per chunk.

I think this option could help in any case where someone is opening data from a Zarr store (the reason we had this opener task) or a netCDF file.

The value of the kwarg should be kept optional because in theory inlining is a tradeoff between fewer tasks and more memory use, but I think there might be a case for setting the default to be True?

Questions: 1) How should I test this? 2) Should it default to False or True? 3) inline_array or inline? (inline_array doesn't really make sense for open_dataset, which creates multiple arrays)

• [x] Closes #1895
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

@rabernat @jbusecke

Also adds some explicit tests (and type hinting) for Variable.chunk(), as I don't think it had dedicated tests before.

• [x] Closes #6459
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Allows for dropping duplicates over multiple dims at once, and adds Dataset.drop_duplicates.

• [x] Inspired by this discussion question
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

These were due to be removed post-0.19.

• [x] Completes deprecation cycle started in #5256
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

@pydata/xarray the release notes for your approval

5889

=============== ====== ==== Package Old New =============== ====== ==== cartopy 0.17 0.18 cftime 1.1 1.2 dask 2.15 2.30 distributed 2.15 2.30 hdf5 1.10 1.12 lxml 4.5 4.6 matplotlib-base 3.2 3.3 numba 0.49 0.51 numpy 1.17 1.18 pandas 1.0 1.1 pint 0.15 0.16 scipy 1.4 1.5 seaborn 0.10 0.11 sparse 0.8 0.11 toolz 0.10 0.11 zarr 2.4 2.5 =============== ====== ====

• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Also reorganised the logic that deals with combining mixed sets of objects (i.e. named dataarrays, unnamed dataarrays, datasets) that was added in #4696.

TODO - same reorganisation / testing but for combine_nested as well as combine_by_coords. EDIT: I'm going to do this in a separate PR, so that this bugfix can be merged without it.

• [x] Closes #5833
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

Follows on from https://github.com/pydata/xarray/pull/5561 by replacing .values with .to_numpy() in more places in the plotting code. This allows pint.Quantity arrays to be plotted without issuing a UnitStrippedWarning (and will generalise better to other duck arrays later).

I noticed the need for this when trying out this example (but trying it without the .dequantify() call first).

(@Illviljan in theory .values should be replaced with .to_numpy() everywhere in the plotting code by the way)

• [ ] Closes #xxxx
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Rectifies the the issue in #5843 by making DataArray.chunks and Variable.chunks consistent with Dataset.chunks. This would obviously need a deprecation cycle before it were merged.

Currently a WIP - I changed the behaviour but this obviously broke quite a few tests and I haven't looked at them yet.

• [x] Closes #5843
• [ ] Tests added
• [x] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Inspired by confusion in #5833, this PR slightly clarifies the error thrown when the user attempts to do ds['var'] = xr.Dataset. The original error is TypeError: cannot directly convert an xarray.Dataset into a numpy array. Instead, create an xarray.DataArray first, either with indexing on the Dataset or by invoking the `to_array()` method. while the new error is TypeError: Cannot assign a Dataset to a single key - only a DataArray or Variable object can be stored under a single key.

• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• ~~New functions/methods are listed in api.rst~~

The default behaviour of da.roll() caught me out whilst trying to hand-write a diff function, so I completed the transition to defaulting to roll_coords=False as the default. It's been throwing a warning for 3 years so I think it's time!

I also improved the docstrings and added type hints whilst there, although mypy doesn't seem to like some of the type hinting :/

• [x] Completes deprecation started in #2360
• [x] Tests updated
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Noticed this hadn't been completed in https://github.com/pydata/xarray/discussions/5659

• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Added type hints to combine_nested and combine_by_coords.

Builds on #4696 because that PR generalised the argument types to include DataArrays, but I couldn't see that branch in the list to base this PR off of.

The "nested list-of-lists" argument to combine_nested opens up a can of worms: the only way I can see to specify the type of a nested list of arbitrary depth is to define the type recursively, but mypy does not currently support recursive type definitions, though some other type checkers can, e.g. Microsoft's Pylance does. We're going to have the same problem when specifying types for open_mfdataset. For now this problem is just ignored by the type checker, meaning that we don't actually check the type of the nested-list-of-lists.

• [x] Passes pre-commit run --all-files
• [ ] ~~User visible changes (including notable bug fixes) are documented in whats-new.rst~~

Release notes:

rst This release brings improvements to plotting of categorical data, the ability to specify how attributes are combined in xarray operations, a new high-level :py:func:`unify_chunks` function, as well as various deprecations, bug fixes, and minor improvements.

• [x] Closes #5588

Stops you needing to call .pint.dequantify() before plotting.

Builds on top of #5568, so that should be merged first.

• [x] Closes (1) from https://github.com/pydata/xarray/issues/3245#issue-484240082
• [x] Tests added
• [x] Tests passing
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] Closes #3245
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

• [x] Closes #2686 and #1839
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

This PR was based off of #3926, though it probably doesn't need to be and could be rebased if we wanted to merge this first.

• [x] Closes #3774
• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

This is an attempt at a proof-of-principle for making animated plots in the way I suggested in #2355. (Also relevant for #2030.)

This example code: ```python import matplotlib.pyplot as plt import xarray as xr

Load data as done in plotting tutorial

airtemps = xr.tutorial.open_dataset('air_temperature') air = airtemps.air - 273.15 air.attrs = airtemps.air.attrs air.attrs['units'] = 'deg C'

Downsample to make reasonably-sized gif

data = air.isel(lat=10, time=slice(None,None,40))

Create animated plot

anim = data.plot(animate_over='time') anim.save('line1.gif', writer='imagemagick') plt.show() ``` now produces this gif: ~~The units on the timeline are formatted incorrectly because this PR isn't merged yet~~

I think it looks pretty good! It even animates the title properly. The actual animation creation only takes one line to do.

This currently only works for a plot with a single line, which is animated over a coordinate dimension. ~~It also required some minor modifications/bugfixes to animatplot, so it probably isn't reproducible right out of the box yet.~~ If you want to try this out then use the develop branch of my forked version of animatplot.

The reason I've put this up is because I wanted to

1. show people the level of complexity required, and
2. get people's opinion on the implementation.

I feel like although it required only ~100 lines extra to do this then the logic is very fragmented and scattered through the plot.line and plot._infer_line_data functions. In 2D this would get even more complicated, but I can't see a good way to abstract the case of animation out?

(@t-makaro I expect you will be interested in this)

EDIT: To-Do list:

• [x] Animate single line
• [x] Animated line and static line on same axes
• [x] Animate multiple lines on same axes
• [x] Multiple animated line plots on same figure
• [ ] ~~FacetGrids of multiple animated lines~~ (will leave for a later PR)
• [ ] Complete set of tests
• [x] Add animatplot as optional dependency
• [x] Add new CI tests using animatplot
• [ ] New documentation page
• [x] Fix issues with formatting of timeline label (fixed by https://github.com/t-makaro/animatplot/pull/46)

Initial work on integrating the multi-dimensional dask-powered histogram functionality from xhistogram into xarray. Just working on the skeleton to fit around the histogram algorithm for now, to be filled in later.

• [ ] Closes #4610
• [x] API skeleton
• [x] Redirect plot.hist
• [ ] Tests added
• [ ] Type hinting
• [ ] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

EDIT: Didn't notice that using git commit --amend has polluted the git history for this branch...

I noticed that the apply_ufunc tutorial notebook says that xarray.apply_ufunc uses dask.array.blockwise, but that's no longer true as of PR #4060 .

• [x] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

Updated the release guide to account for what is now automated via github actions, and any other bits I felt could be clearer.

Now only 16 easy steps!

• Motivated by #5232 and #5244
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Inspired by a question asked in #4696, but does not close that issue - [x] Passes pre-commit run --all-files - [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Does this need a test?

• [x] Closes #5073
• [ ] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Changes error introduced in #5231 into a warning, as discussed.

Fixes bug which allowed incorrect arguments to be passed to open_mfdataset without complaint.

The combination open_mfdataset(files, combine='by_coords', concat_dim='t') should never have been permitted, and in fact it wasn't permitted until the last part of the deprecation process from the old auto_combine. It makes no sense to pass this combination because the combine_by_coords function does not have a concat_dim argument at all!

The effect was pretty benign - the concat_dim arg wasn't really used for anything in that case, and the result of passing dodgy datasets would just be a less informative error. However there were multiple tests which assumed this behaviour was okay - I had to remove that particular parametrization for a bunch of your join tests @dcherian because they now fail with a different (clearer) error.

I also noticed a related issue which I fixed - internally open_mfdataset was performing a rearrangement of the input datasets that it needs for the case combine='nested', even in the case combine='by_coords'. I hadn't previously realised that we can just skip this rearrangement without issue, so open_mfdataset(combine='by_coords') should be a little bit faster now.

• [x] Closes #5230
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] Closes #3992
• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [x] Finishes deprecation cycle started in #2616 (was supposed to have been done in 0.15)
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

I've set combine='by_coords' as the default argument to open_mfdataset. Technically we could go for either, as the deprecation warning just told users to make it explicit from now on, but going for by_coords rather than nested means that: - The concat_dim argument is not needed by default, - The default behaviour of the function is the "magic" one - users have to opt-in to the more explicit behaviour.

dataarray.interpolate_na was dropping attrs because even though the internal apply_ufunc call was being passed keep_attrs=True, the order of arguments index, da to apply_ufunc meant that it was trying to keep only the non-existent attrs from the first argument. I just swapped them round, and added a globally-aware keep_attrs kwarg.

• [x] Closes #3968
• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API