Builds on #8997

Expose Grouper and Resampler as public API

TODO: - [ ] Consider avoiding IndexVariable

• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

What happened?

Found by hypothesis ``` import xarray as xr import numpy as np

var = xr.Variable(dims="2", data=np.array(['1970-01-01T00:00:00.000000000', '1970-01-01T00:00:00.000000002', '1970-01-01T00:00:00.000000001'], dtype='datetime64[ns]')) var1 = xr.Variable(data=np.array([0], dtype=np.uint32), dims=['1'], attrs={})

state = xr.Dataset() state['2'] = var state = state.stack({"0": ["2"]}) state['1'] = var1 state['1_'] = var1#.copy(deep=True) state = state.swap_dims({"1": "1_"}) xr.testing.assertions._assert_internal_invariants(state, False) ```

This swaps simple pandas indexed dims, but the multi-index that is in the dataset and not affected by the swap_dims op ends up broken.

cc @benbovy

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [ ] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Not for merging, just proof that it can be done nicely :)

Now builds on #8840 ~Builds on an older version of #8507~

Try it out!

```python import xarray as xr from xarray.core.groupers import SeasonGrouper, SeasonResampler

ds = xr.tutorial.open_dataset("air_temperature")

custom seasons!

ds.air.groupby(time=SeasonGrouper(["JF", "MAM", "JJAS", "OND"])).mean()

ds.air.resample(time=SeasonResampler(["DJF", "MAM", "JJAS", "ON"])).count() ```

Discussed in https://github.com/pydata/xarray/discussions/8567

The new asv version comes with decorators for parameterizing and skipping, and the ability to use mamba to create environments.

https://github.com/airspeed-velocity/asv/releases

https://asv.readthedocs.io/en/v0.6.1/writing_benchmarks.html#skipping-benchmarks

This might help us reduce benchmark times a bit, or at least simplify the code some.

Use as_indexable instead of NumpyIndexingAdapter

What is your issue?

Copying over @max-sixty's comment from https://github.com/pydata/xarray/pull/8304#discussion_r1358873522

I haven't been in touch with the NameArray discussions so forgive a glib comment — but re https://github.com/pydata/xarray/issues/3891 — this would be a "once-in-a-library" opportunity to always retain attrs in aggregations, removing the keep_attrs option in methods.

(Xarray could still handle them as it wished, so xarray's external interface wouldn't need to change immediately...)

@pydata/xarray Should we just delete the keep_attrs kwarg completely for NamedArray and always propagate attrs? obj.attrs.clear() seems just as easy to type.

• [x] Closes #8090
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

ideally GroupBy._infer_concat_args() would return a xr.Coordinates object that contains both the coordinate(s) and their (multi-)index to assign to the result (combined) object.

The goal is to avoid calling create_default_index_implicit(coord) below where coord is a pd.MultiIndex or a single IndexVariable wrapping a multi-index. If coord is a Coordinates object, we could do combined = combined.assign_coords(coord) instead.

https://github.com/pydata/xarray/blob/e2b6f3468ef829b8a83637965d34a164bf3bca78/xarray/core/groupby.py#L1573-L1587

There are actually more general issues:

• The group parameter of Dataset.groupby being a single variable or variable name, it won't be possible to do groupby on a full pandas multi-index once we drop its dimension coordinate (#8143). How can we still support it? Maybe passing a dimension name to group and check that there's only one index for that dimension?
• How can we support custom, multi-coordinate indexes with groupby? I don't have any practical example in mind, but in theory just passing a single coordinate name as group will invalidate the index. Should we drop the index in the result? Or, like suggested above pass a dimension name as group and check the index?

Originally posted by @benbovy in https://github.com/pydata/xarray/issues/8140#issuecomment-1709775666

Is your feature request related to a problem?

flox recently added support for first, last, nanfirst, nanlast. So we should support that on the Xarray GroupBy object.

• [x] Tests added

Fixes yet another bug with GroupBy reductions. We weren't assigning the group index when reducing by a subset of dimensions present on the grouper

This will only pass when flox 0.7.1 reaches conda-forge.

Is your feature request related to a problem?

Scipy 1.10.0 has some great improvements to interpolation (release notes) particularly around the fancier methods like pchip.

It'd be good to see if we can simplify some of our code (or even enable using these options).

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

• [x] Closes #4288
• [ ] Tests added
• [x] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Refactors line plotting to use a _plot1d decorator.

Next i'll use this decorator on hist so we can "facet" and "hue" histograms.

see #4288

What is your issue?

File roundtripping tests in test_distributed.py don't use the roundtrip context manager (thpugh one uses create_tmp_file) so I don't think any created files are being cleaned up.

Example: https://github.com/pydata/xarray/blob/09e467a6a3a8ed68c6c29647ebf2b09288145da1/xarray/tests/test_distributed.py#L91-L119

What is your issue?

From @benbovy in https://github.com/pydata/xarray/pull/6477

• extract common indexes and explicitly pass them to the Dataset and DataArray constructors (when implemented) that are called in the broadcast helper functions (there are some temporary and ugly hacks in create_default_index_implicit so that it works now with pandas multi-indexes wrapped in coordinate variables without the need to pass those indexes explicitly)
• extract common indexes based on the dimension(s) of their coordinates and not their name (e.g., case of non-dimension but indexed coordinate)

What happened?

python ds = xr.Dataset( {"a": ("x", [1, 2, 3])}, coords={"c": ("x", [1, 2, 3]), "d": ("y", [1, 2, 3, 4])} ) print(ds.coords) mapped = ds.map(lambda x: x) print(mapped.coords)

Variables d gets dropped in the map call. It does not share any dimensions with any of the data variables. Coordinates: c (x) int64 1 2 3 d (y) int64 1 2 3 4 Coordinates: c (x) int64 1 2 3

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

Relevant log output

No response

Anything else we need to know?

No response

Environment

xarray 2022.03.0

What happened?

From https://github.com/pydata/xarray/discussions/6370

Calling apply_ufunc(..., dask="parallelized") with no core dimensions and dask input "works" but raises an error on compute (ValueError: axes don't match array from np.transpose).

python xr.apply_ufunc( lambda x: np.mean(x), dt, dask="parallelized" )

What did you expect to happen?

With numpy data the apply_ufunc call does raise an error:

xr.apply_ufunc( lambda x: np.mean(x), dt.compute(), dask="parallelized" )

ValueError: applied function returned data with unexpected number of dimensions. Received 0 dimension(s) but expected 1 dimensions with names: ('x',)

Minimal Complete Verifiable Example

``` python import xarray as xr

dt = xr.Dataset( data_vars=dict( value=(["x"], [1,1,2,2,2,3,3,3,3,3]), ), coords=dict( lon=(["x"], np.linspace(0,1,10)), ), ).chunk(chunks={'x': tuple([2,3,5])}) # three chunks of different size

xr.apply_ufunc( lambda x: np.mean(x), dt, dask="parallelized" ) ```

Relevant log output

No response

Anything else we need to know?

No response

Environment

N/A

I noticed that we don't test resampling with dask arrays (well just one).

This could be a good opportunity to convert test_groupby.py to use test fixtures like in https://github.com/pydata/xarray/pull/5411

From https://github.com/pydata/xarray/pull/5931

It would be good to add a test checking that custom backend entrypoints work. This might involve creating a dummy package that registers an entrypoint (https://github.com/pydata/xarray/pull/5931#issuecomment-959131968)

Recent dask version support assigning to a list of ints along one dimension. we can use this for unstacking (diff builds on #5577)

```diff diff --git i/xarray/core/variable.py w/xarray/core/variable.py index 222e8dab9..a50dfc574 100644 --- i/xarray/core/variable.py +++ w/xarray/core/variable.py @@ -1593,11 +1593,9 @@ class Variable(AbstractArray, NdimSizeLenMixin, VariableArithmetic): else: dtype = self.dtype

• if sparse:
• if sparse and not is_duck_dask_array(reordered): # unstacking a dense multitindexed array to a sparse array

• Use the sparse.COO constructor until sparse supports advanced indexing

• https://github.com/pydata/sparse/issues/114

• TODO: how do we allow different sparse array types

• Use the sparse.COO constructor since we cannot assign to sparse.COO

       from sparse import COO
  
       codes = zip(*index.codes)
  

  @@ -1618,19 +1616,23 @@ class Variable(AbstractArray, NdimSizeLenMixin, VariableArithmetic): )

   else:
  

  • dask supports assigning to a list of ints along one axis only.

  • So we construct an array with the last dimension flattened,

  • assign the values, then reshape to the final shape.

  • intermediate_shape = reordered.shape[:-1] + (np.prod(new_dim_sizes),)
  • indexer = np.ravel_multi_index(index.codes, new_dim_sizes) data = np.full_like( self.data, fill_value=fill_value,
  • shape=new_shape,

  • shape=intermediate_shape, dtype=dtype, )

     # Indexer is a list of lists of locations. Each list is the locations
     # on the new dimension. This is robust to the data being sparse; in that
     # case the destinations will be NaN / zero.
    

    • sparse doesn't support item assigment,

    • https://github.com/pydata/sparse/issues/114

    • data[(..., *indexer)] = reordered
    • data[(..., indexer)] = reordered
    • data = data.reshape(new_shape)

    return self._replace(dims=new_dims, data=data) ```

This should be what alignment.reindex_variables is doing but I don't fully understand that function.

The annoying bit is figuring out when to use this version and what to do with things like dask wrapping sparse. I think we want to loop over each variable in Dataset.unstack calling Variable.unstack and dispatch based on the type of Variable.data to easily handle all the edge cases.

cc @Illviljan if you're interested in implementing this

This is such a common question that we should address it in the docs.

Just saying that some hdf5 files can be opened with h5netcdf, and that the user needs to manually create xarray objects with everything else should be enough.

https://xarray.pydata.org/en/stable/io.html

Can we remove xarray.core.rolling and core.rolling on the left and right respectively? I think the API reference would be a lot more readable if we could do that

Is your feature request related to a problem? Please describe. .encoding is somewhat hidden since we don't show it in a repr.

Describe the solution you'd like I think it'd be nice to add it to the HTML repr, collapsed by default.

We should list supported options for backend_kwargs in the docstring for open_datasetand possibly in io.rst

xref #4187

What do people think of adding ReviewNB https://www.reviewnb.com/ to facilitate easy reviewing of example notebooks?

It would be nice to have a separate page that detailed this for users unfamiliar with netCDF.