xref #8509, #6610

Rendered version here

@pydata/xarray I've been poking at this on and off for a year now and finally figured out how to do it cleanly (#8509).

I wrote up a design doc for Grouper objects that allow custom conversion of DataArrays to integer group codes, following the NEP template (which is absolutely great!). Such Grouper objects allow us to generalize the GroupBy interface to a much larger class of problems, and eventually provide a nice path to grouping by multiple variables (#6610)

8509 implements two custom Groupers for you to try out :)

```python import xarray as xr from xarray.core.groupers import SeasonGrouper, SeasonResampler

ds = xr.tutorial.open_dataset("air_temperature")

custom seasons!

ds.air.groupby(time=SeasonGrouper(["JF", "MAM", "JJAS", "OND"])).mean()

ds.air.resample(time=SeasonResampler(["DJF", "MAM", "JJAS", "ON"])).count() ```

All comments are welcome, 1. there are a couple of specific API and design decisions to be made. I'll make some comments pointing these out. 2. I'm also curious about what Grouper objects we should provide in Xarray.

cc @ilan-gold @ivirshup @aulemahal @tomvothecoder @jbusecke @katiedagon - it would be good to hear what "Groupers" would be useful for your work / projects. I bet you already have examples that fit this proposal

• [x] fix breaking attrs test
• [x] Look at numeric_only in NAMED_ARRAY_OBJECT
• xref #8238
• [ ] Tests added

We've accumulated a nice number of changes.

Can someone volunteer to do a release in the next few days?

Implements most of https://github.com/pydata/xarray/issues/4325#issuecomment-716399575 ``` python %load_ext memory_profiler

import numpy as np import xarray as xr

temp = xr.DataArray(np.zeros((5000, 500)), dims=("x", "y"))

roll = temp.rolling(x=10, y=20)

%memit roll.sum() %memit roll.reduce(np.sum) %memit roll.reduce(np.nansum) # master branch behaviour peak memory: 245.18 MiB, increment: 81.92 MiB peak memory: 226.09 MiB, increment: 62.69 MiB peak memory: 4493.82 MiB, increment: 4330.43 MiB `` - [x] xref #4325 - [x] asv benchmarks added - [x] Passespre-commit run --all-files- [x] User visible changes (including notable bug fixes) are documented inwhats-new.rst`

Is your feature request related to a problem?

Given all the outreachy interest (and perhaps just in general) it may be nice to enable a welcome bot like on the Jupyter repos

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

This is motivated by https://docs.rapids.ai/api/kvikio/stable/api.html#kvikio.zarr.GDSStore which on read loads the data directly into GPU memory.

Currently we rely on np.asarray to convert a BackendArray wrapped with a number of lazy indexing classes to a real array but this breaks for GDSStore because the underlying array is a cupy array, so using np.asarray raises an error.

np.asarray will raise if a non-numpy array is returned so we need to use something else.

Here I added get_array which like np.array recurses down until it receives a duck array.

Quite a few things are broken I think , but I'd like feedback on the approach.

I considered np.asanyarray(..., like=...) but that would require the lazy indexing classes to know what they're wrapping which doesn't seem right.

Ref: https://github.com/xarray-contrib/cupy-xarray/pull/10 which adds a kvikio backend entrypoint

Minimal Complete Verifiable Example:

Here's a small array with too-small chunk sizes just as an example ```python

Put your MCVE code here

import dask.array import xarray as xr

var = xr.Variable(("x", "y", "z"), dask.array.random.random((4, 18483, 1000), chunks=(1, 183, -1))) ```

Now stack two dimensions, this is a 100x increase in chunk size (in my actual code, 85MB chunks become 8.5GB chunks =) )

var.stack(new=("x", "y"))

But calling reshape on the dask array preserves the original chunk size var.data.reshape((4*18483, -1))

Solution

Ah, found it , we transpose then reshape in Variable_stack_once. https://github.com/pydata/xarray/blob/f915515d610b4471888fa44dfb00dbae3fd22349/xarray/core/variable.py#L1521-L1527

Writing those steps with pure dask yields the same 100x increase in chunksize

python var.data.transpose([2, 0, 1]).reshape((-1, 4*18483))

Anything else we need to know?:

Environment:

What happened?

Seen over in cf_xarray

Using xr.core.resample.Resample worked prior to https://github.com/pydata/xarray/pull/6702.

Now we need to use from xarray.core.resample import Resample

I don't know if this is something that needs to be fixed or only worked coincidentally earlier. But I thought it was worth discussing prior to release.

Thanks to @aulemahal for spotting

Is your feature request related to a problem?

From @keewis in #6645

if anyone can figure out how to create PEP440 (and thus PyPI) compatible development versions I think we can have a CI publish every commit on main to TestPyPI.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

This PR makes a number of updates to api.rst. The main goal is to make it easier to navigate using the right-side "Contents" menu. By moving things from api-hidden.rst to api.rst many methods are now easier-to-discover.

list all reduction methods in their own subsection under "Dataset" and "DataArray".

before

after

List reductions for groupby etc

I've also added sections for rolling, coarsen, groupby, resample that again list available reductions. These were previously hidden

Class method names are much more readable.

Before

after

The current RTD build is failing in plotting.rst

`` sphinx.errors.SphinxParallelError: RuntimeError: Non Expected exception in/home/docs/checkouts/readthedocs.org/user_builds/xray/checkouts/latest/doc/user-guide/plotting.rst` line None

Sphinx parallel build error: RuntimeError: Non Expected exception in /home/docs/checkouts/readthedocs.org/user_builds/xray/checkouts/latest/doc/user-guide/plotting.rst line None [IPKernelApp] WARNING | Parent appears to have exited, shutting down. ```

• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Just like the pandas method.

RTD now supports using mamba instead of conda using a feature flag : https://docs.readthedocs.io/en/latest/guides/feature-flags.html#available-flags

Maybe it's worth seeing if this will reduce docs build time on CI?

Closes #3490 Closes #4065 Closes #3433 Closes #3595

• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Still needs work but I thought I'd get opinions on defaults and the approach.

When rechunking by the current chunk size, name should not change. Add a __dask_tokenize__ method for ReprObject so that this behaviour is present when DataArrays are converted to temporary Datasets and back.

• [x] Closes #3350
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Our examples now have binder links! e.g. https://xarray.pydata.org/en/latest/examples/ERA5-GRIB-example.html

but this is suboptimal because we aren't defining an environment so the only things that get installed are the packages that xarray requires : numpy & pandas. We need to at least add cfgrid, matplotlib, cftime, netCDF4 to get all the examples to work out of the box.

• [x] Closes #3161
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [x] Closes #3440

This PR makes DataArray.__init__ ignore the pandas attrs.

Not sure what we want to do about these attributes in the long term. One option would be to pop the name attribute, assign to DataArray.name and keep the rest as DataArray.attrs? But what if name clashes with the provided name?

• [x] Closes #1756 (remove if there is no corresponding issue, which should only be the case for minor changes)
• [x] Tests added (for all bug fixes or enhancements)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

• [x] Closes #2885
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

I'm seeing this warning when using open_mfdataset(list_of_files) on master. /gpfs/u/home/dcherian/python/xarray/xarray/backends/api.py:783: FutureWarning: In xarray version 0.13 `auto_combine` will be deprecated. /gpfs/u/home/dcherian/python/xarray/xarray/backends/api.py:783: FutureWarning: The datasets supplied have global dimension coordinates. You may want to use the new `combine_by_coords` function (or the `combine='by_coords'` option to `open_mfdataset` to order the datasets before concatenation. Alternatively, to continue concatenating based on the order the datasets are supplied in in future, please use the new `combine_nested` function (or the `combine='nested'` option to open_mfdataset).

I don't think we should be issuing this warning when using open_mfdataset with default kwargs. Also there's a typo: "in in future"

Seems like we're close to a release with some nice features. The following PRs are either done or are really close:

Features:

• [ ] #1597 stack/unstacking for ML
• [x] #2556 transpose_coords
• [x] #2616 ND combine
• [x] #2964 fill_value for concat / combine
• [ ] #2277 dataset plotting needs a review.

indexes:

• [ ] #2816
• [x] #2859 consistency checks

Bugfixes:

• [x] #2207 chunksizes
• [x] #2941
• [x] #2965
• [x] #2968
• [x] #2971
• [x] #2973
• [ ] #2942 needs some help.
• [x] #2924

Uses attrs long_name/standard_name, units if available.

• [x] Closes #2135, #1630, #1787
• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

This PR basically uses @rabernat's label_from_attrs in #2135.

Examples:

I also changed the way histograms are labelled Old:

New: