xref #8509, #6610

Rendered version here

@pydata/xarray I've been poking at this on and off for a year now and finally figured out how to do it cleanly (#8509).

I wrote up a design doc for Grouper objects that allow custom conversion of DataArrays to integer group codes, following the NEP template (which is absolutely great!). Such Grouper objects allow us to generalize the GroupBy interface to a much larger class of problems, and eventually provide a nice path to grouping by multiple variables (#6610)

8509 implements two custom Groupers for you to try out :)

```python import xarray as xr from xarray.core.groupers import SeasonGrouper, SeasonResampler

ds = xr.tutorial.open_dataset("air_temperature")

custom seasons!

ds.air.groupby(time=SeasonGrouper(["JF", "MAM", "JJAS", "OND"])).mean()

ds.air.resample(time=SeasonResampler(["DJF", "MAM", "JJAS", "ON"])).count() ```

All comments are welcome, 1. there are a couple of specific API and design decisions to be made. I'll make some comments pointing these out. 2. I'm also curious about what Grouper objects we should provide in Xarray.

cc @ilan-gold @ivirshup @aulemahal @tomvothecoder @jbusecke @katiedagon - it would be good to hear what "Groupers" would be useful for your work / projects. I bet you already have examples that fit this proposal

• [x] fix breaking attrs test
• [x] Look at numeric_only in NAMED_ARRAY_OBJECT
• xref #8238
• [ ] Tests added

We've accumulated a nice number of changes.

Can someone volunteer to do a release in the next few days?

I think a good "infrastructure" target for the NASA OSS call would be to expand our benchmarking suite (https://pandas.pydata.org/speed/xarray/#/)

AFAIK running these in a useful manner on CI is still unsolved (please correct me if I'm wrong). But we can always run it on an NCAR machine using a cron job.

Thoughts?

cc @scottyhq

A quick survey of work needed (please append): - [ ] indexing & slicing #3382 #2799 #2227 - [ ] DataArray construction #4744 - [ ] attribute access #4741, #4742 - [ ] property access #3514 - [ ] reindexing? https://github.com/pydata/xarray/issues/1385#issuecomment-297539517 - [x] alignment #3755, #7738 - [ ] assignment #1771 - [ ] coarsen - [x] groupby #659 #7795 #7796 - [x] resample #4498 #7795 - [ ] weighted #4482 #3883 - [ ] concat #7824 - [ ] merge - [ ] open_dataset, open_mfdataset #1823 - [ ] stack / unstack - [ ] apply_ufunc? - [x] interp #4740 #7843 - [ ] reprs #4744 - [x] to_(dask)_dataframe #7844 #7474

Related: #3514

What is your issue?

flox supports grouping by multiple variables (would fix #324, #1056) and grouping by dask variables (would fix #2852).

To enable this in GroupBy we need to update the constructor's signature to 1. Accept multiple "by" variables. 2. Accept "expected group labels" for grouping by dask variables (like bins for groupby_bins which already supports grouping by dask variables). This lets us construct the output coordinate without evaluating the dask variable. 3. We may also want to simultaneously group by a categorical variable (season) and bin by a continuous variable (air temperature). So we also need a way to indicate whether the "expected group labels" are "bin edges" or categories.

The signature in flox is (may be errors!) python xarray_reduce( obj: Dataset | DataArray, *by: DataArray | str, func: str | Aggregation, expected_groups: Sequence | np.ndarray | None = None, isbin: bool | Sequence[bool] = False, ... )

You would calculate that last example using flox as python xarray_reduce( ds, "season", "air_temperature", expected_groups=[None, np.arange(21, 30, 1)], isbin=[False, True], ... )

The use of expected_groups and isbin seems ugly to me (the names could also be better!)

I propose we update groupby's signature to 1. change group: DataArray | str to group: DataArray | str | Iterable[str] | Iterable[DataArray] 2. We could add a top-level xr.Bins object that wraps bin edges + any kwargs to be passed to pandas.cut. Note our current groupby_bins signature has a bunch of kwargs passed directly to pandas.cut. 3. Finally add groups: None | ArrayLike | xarray.Bins | Iterable[None | ArrayLike | xarray.Bins] to pass the "expected group labels". 1. If None, then groups will be auto-detected from non-dask group arrays (if None for a dask group, then raise error). 1. If xarray.Bins indicates binning by the appropriate variables 1. If ArrayLike treat as categorical. 1. groups is a little too similar to group so we should choose a better name. 1. The ordering of ArrayLike would let us fix #757 (pass the seasons in the order you want them in the output)

So then that example becomes python ds.groupby( ["season", "air_temperature"], # season is numpy, air_temperature is dask groups=[None, xr.Bins(np.arange(21, 30, 1), closed="right")], )

Thoughts?

Implements most of https://github.com/pydata/xarray/issues/4325#issuecomment-716399575 ``` python %load_ext memory_profiler

import numpy as np import xarray as xr

temp = xr.DataArray(np.zeros((5000, 500)), dims=("x", "y"))

roll = temp.rolling(x=10, y=20)

%memit roll.sum() %memit roll.reduce(np.sum) %memit roll.reduce(np.nansum) # master branch behaviour peak memory: 245.18 MiB, increment: 81.92 MiB peak memory: 226.09 MiB, increment: 62.69 MiB peak memory: 4493.82 MiB, increment: 4330.43 MiB `` - [x] xref #4325 - [x] asv benchmarks added - [x] Passespre-commit run --all-files- [x] User visible changes (including notable bug fixes) are documented inwhats-new.rst`

Is your feature request related to a problem?

Given all the outreachy interest (and perhaps just in general) it may be nice to enable a welcome bot like on the Jupyter repos

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

This is motivated by https://docs.rapids.ai/api/kvikio/stable/api.html#kvikio.zarr.GDSStore which on read loads the data directly into GPU memory.

Currently we rely on np.asarray to convert a BackendArray wrapped with a number of lazy indexing classes to a real array but this breaks for GDSStore because the underlying array is a cupy array, so using np.asarray raises an error.

np.asarray will raise if a non-numpy array is returned so we need to use something else.

Here I added get_array which like np.array recurses down until it receives a duck array.

Quite a few things are broken I think , but I'd like feedback on the approach.

I considered np.asanyarray(..., like=...) but that would require the lazy indexing classes to know what they're wrapping which doesn't seem right.

Ref: https://github.com/xarray-contrib/cupy-xarray/pull/10 which adds a kvikio backend entrypoint

Minimal Complete Verifiable Example:

Here's a small array with too-small chunk sizes just as an example ```python

Put your MCVE code here

import dask.array import xarray as xr

var = xr.Variable(("x", "y", "z"), dask.array.random.random((4, 18483, 1000), chunks=(1, 183, -1))) ```

Now stack two dimensions, this is a 100x increase in chunk size (in my actual code, 85MB chunks become 8.5GB chunks =) )

var.stack(new=("x", "y"))

But calling reshape on the dask array preserves the original chunk size var.data.reshape((4*18483, -1))

Solution

Ah, found it , we transpose then reshape in Variable_stack_once. https://github.com/pydata/xarray/blob/f915515d610b4471888fa44dfb00dbae3fd22349/xarray/core/variable.py#L1521-L1527

Writing those steps with pure dask yields the same 100x increase in chunksize

python var.data.transpose([2, 0, 1]).reshape((-1, 4*18483))

Anything else we need to know?:

Environment:

I wrote this little notebook today while trying to get some satellite data in form that was nice to work with: https://gist.github.com/dcherian/66269bc2b36c2bc427897590d08472d7

I think it would make a useful example for the docs.

A few questions: 1. Do you think it'd be a good addition to the examples? 2. Is this the recommended way of adding meaningful co-ordinates, expanding dims etc.? The main bit is this function: ``` def preprocess(ds):

    dsnew = ds.copy()
    dsnew['latitude'] = xr.DataArray(np.linspace(90, -90, 180),
                                     dims=['phony_dim_0'])
    dsnew['longitude'] = xr.DataArray(np.linspace(-180, 180, 360),
                                      dims=['phony_dim_1'])
    dsnew = (dsnew.rename({'l3m_data': 'sss',
                           'phony_dim_0': 'latitude',
                           'phony_dim_1': 'longitude'})
             .set_coords(['latitude', 'longitude'])
             .drop('palette'))

    dsnew['time'] = (pd.to_datetime(dsnew.attrs['time_coverage_start'])
                     + np.timedelta64(3, 'D') + np.timedelta64(12, 'h'))
    dsnew = dsnew.expand_dims('time').set_coords('time')

    return dsnew

```

Also open to other feedback...

What happened?

I'm calling cupy.round on a DataArray wrapping a cupy array and it raises an error here: https://github.com/pydata/xarray/blob/3f7cc2da33d81e76afbfb82da57143b624b03a88/xarray/core/common.py#L155-L156

Traceback below:

What did you expect to happen?

Not an error? I'm not sure what's expected

np.round(dataarray) does actually work successfully.

My question is : Do we need to update AbstractArray.__array__ to return the underlying duck array instead of always a numpy array?

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

What happened?

Seen over in cf_xarray

Using xr.core.resample.Resample worked prior to https://github.com/pydata/xarray/pull/6702.

Now we need to use from xarray.core.resample import Resample

I don't know if this is something that needs to be fixed or only worked coincidentally earlier. But I thought it was worth discussing prior to release.

Thanks to @aulemahal for spotting

Is your feature request related to a problem?

From @keewis in #6645

if anyone can figure out how to create PEP440 (and thus PyPI) compatible development versions I think we can have a CI publish every commit on main to TestPyPI.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Currently pad adds NaNs for coordinate labels, which results in substantially reduced functionality.

We need to think about 1. Int, Float, Datetime64, CFTime indexes: linearly extrapolate? Should we care whether the index is sorted or not? (I think not) 2. MultiIndexes: ?? 3. CategoricalIndexes: ?? 4. Unindexed dimensions

EDIT: Added unindexed dimensions

This PR makes a number of updates to api.rst. The main goal is to make it easier to navigate using the right-side "Contents" menu. By moving things from api-hidden.rst to api.rst many methods are now easier-to-discover.

list all reduction methods in their own subsection under "Dataset" and "DataArray".

before

after

List reductions for groupby etc

I've also added sections for rolling, coarsen, groupby, resample that again list available reductions. These were previously hidden

Class method names are much more readable.

Before

after

The current RTD build is failing in plotting.rst

`` sphinx.errors.SphinxParallelError: RuntimeError: Non Expected exception in/home/docs/checkouts/readthedocs.org/user_builds/xray/checkouts/latest/doc/user-guide/plotting.rst` line None

Sphinx parallel build error: RuntimeError: Non Expected exception in /home/docs/checkouts/readthedocs.org/user_builds/xray/checkouts/latest/doc/user-guide/plotting.rst line None [IPKernelApp] WARNING | Parent appears to have exited, shutting down. ```

• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Just like the pandas method.

RTD now supports using mamba instead of conda using a feature flag : https://docs.readthedocs.io/en/latest/guides/feature-flags.html#available-flags

Maybe it's worth seeing if this will reduce docs build time on CI?

Closes #3490 Closes #4065 Closes #3433 Closes #3595

• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Still needs work but I thought I'd get opinions on defaults and the approach.

When rechunking by the current chunk size, name should not change. Add a __dask_tokenize__ method for ReprObject so that this behaviour is present when DataArrays are converted to temporary Datasets and back.

• [x] Closes #3350
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Our examples now have binder links! e.g. https://xarray.pydata.org/en/latest/examples/ERA5-GRIB-example.html

but this is suboptimal because we aren't defining an environment so the only things that get installed are the packages that xarray requires : numpy & pandas. We need to at least add cfgrid, matplotlib, cftime, netCDF4 to get all the examples to work out of the box.

• [x] Closes #3161
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [x] Closes #3440

This PR makes DataArray.__init__ ignore the pandas attrs.

Not sure what we want to do about these attributes in the long term. One option would be to pop the name attribute, assign to DataArray.name and keep the rest as DataArray.attrs? But what if name clashes with the provided name?

• [x] Closes #1756 (remove if there is no corresponding issue, which should only be the case for minor changes)
• [x] Tests added (for all bug fixes or enhancements)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

• [x] Closes #2885
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

I'm seeing this warning when using open_mfdataset(list_of_files) on master. /gpfs/u/home/dcherian/python/xarray/xarray/backends/api.py:783: FutureWarning: In xarray version 0.13 `auto_combine` will be deprecated. /gpfs/u/home/dcherian/python/xarray/xarray/backends/api.py:783: FutureWarning: The datasets supplied have global dimension coordinates. You may want to use the new `combine_by_coords` function (or the `combine='by_coords'` option to `open_mfdataset` to order the datasets before concatenation. Alternatively, to continue concatenating based on the order the datasets are supplied in in future, please use the new `combine_nested` function (or the `combine='nested'` option to open_mfdataset).

I don't think we should be issuing this warning when using open_mfdataset with default kwargs. Also there's a typo: "in in future"

Seems like we're close to a release with some nice features. The following PRs are either done or are really close:

Features:

• [ ] #1597 stack/unstacking for ML
• [x] #2556 transpose_coords
• [x] #2616 ND combine
• [x] #2964 fill_value for concat / combine
• [ ] #2277 dataset plotting needs a review.

indexes:

• [ ] #2816
• [x] #2859 consistency checks

Bugfixes:

• [x] #2207 chunksizes
• [x] #2941
• [x] #2965
• [x] #2968
• [x] #2971
• [x] #2973
• [ ] #2942 needs some help.
• [x] #2924

Uses attrs long_name/standard_name, units if available.

• [x] Closes #2135, #1630, #1787
• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

This PR basically uses @rabernat's label_from_attrs in #2135.

Examples:

I also changed the way histograms are labelled Old:

New: