Congratulations @dcherian & @Illviljan & [edit] @andersy005 !!!

and @andersy005 !

👍 this looks great to me!

a global/context option that changes the default value of method

Unfortunately the optimal method depends on distribution of group labels across chunks, so a global option doesn't make sense. It would make sense to create a method="auto" and use that but it doesn't exist yet ("cohorts" is closest)

Thanks for testing it out! I was going to ping xclim when this finally got merged. Presumably you haven't found any bugs?

You can pass method as .mean(..., method=...). Clearly this needs docs :)

We could actually consider adding flox_kwargs to the groupby constructor since a method is really only dependent on the distribution of group labels across the chunks. Right now, I'd just like this to get merged :)

For resampling-type, we are using cohorts by default which generalizes to blockwise when applicable but is slower at graph-construction time. Note you can only blockwise if all members of a group are in a single block. So if you are resampling to yearly but a year of data occupies multiple chunks, you want "cohorts", not "blockwise".

Yay, mypy is passing now after flox updates. :)

Copy/pasteing the Resample-class removes the rest of the errors. Not especially elegant though, maybe there's a better way?

Ok, this kind of works. But there's a new error because I defined the Data*ResampleBase-classes with a for-loop.

Thanks @Illviljan I'm having trouble getting the inheritance order right and keeping mypy happy. Help is very welcome!

There's a bit of mypy errors now. Maybe just removíng _flox_reduce in _reductions.py is enough?

@pydata/xarray This is ready to go. It's mostly one adaptor function and a lot of new tests. It does need docs, I can add that in a future PR.

By default, we use a strategy ("split-reduce") that is very similar to our current one with dask arrays, so users will have to explicitly choose a new strategy to see much improvements. For resampling we can choose a sensible default that should show only improvements, and no regressions ("cohorts")

@pydata/xarray this is blocked by https://github.com/pydata/xarray/issues/6430 but is ready for review.

This is a strict improvement now, well done!

before after ratio [4c865d60] [9b13c974] - 18.7±0.3ms 11.0±0.1ms 0.59 groupby.GroupByDask.time_agg_small_num_groups('sum', 2) - 24.3±1ms 13.8±0.2ms 0.57 groupby.ResampleDask.time_agg_small_num_groups('mean', 2) - 22.0±0.2ms 12.0±0.3ms 0.54 groupby.ResampleDask.time_agg_small_num_groups('mean', 1) - 13.5±0.4ms 7.28±0.3ms 0.54 groupby.GroupByDask.time_agg_small_num_groups('sum', 1) - 33.7±0.4ms 14.2±0.3ms 0.42 groupby.ResampleDask.time_agg_small_num_groups('sum', 2) - 30.7±0.4ms 11.9±0.2ms 0.39 groupby.ResampleDask.time_agg_small_num_groups('sum', 1) - 104±1ms 8.21±0.08ms 0.08 groupby.Resample.time_agg_large_num_groups('sum', 1) - 119±1ms 8.84±0.2ms 0.07 groupby.Resample.time_agg_large_num_groups('sum', 2) - 115±2ms 8.45±0.2ms 0.07 groupby.Resample.time_agg_large_num_groups('mean', 1) - 134±1ms 9.60±0.2ms 0.07 groupby.Resample.time_agg_large_num_groups('mean', 2) - 257±3ms 10.2±0.3ms 0.04 groupby.GroupBy.time_agg_large_num_groups('mean', 2) - 260±5ms 9.99±0.2ms 0.04 groupby.GroupBy.time_agg_large_num_groups('sum', 2) - 190±6ms 6.47±0.2ms 0.03 groupby.GroupBy.time_agg_large_num_groups('mean', 1) - 193±2ms 6.31±0.2ms 0.03 groupby.GroupBy.time_agg_large_num_groups('sum', 1) - 368±5ms 10.9±0.2ms 0.03 groupby.GroupByDask.time_agg_large_num_groups('mean', 2) - 494±2ms 13.7±0.3ms 0.03 groupby.ResampleDask.time_agg_large_num_groups('mean', 2) - 451±8ms 12.3±0.3ms 0.03 groupby.ResampleDask.time_agg_large_num_groups('mean', 1) - 318±3ms 7.13±0.04ms 0.02 groupby.GroupByDask.time_agg_large_num_groups('mean', 1) - 531±6ms 10.6±0.2ms 0.02 groupby.GroupByDask.time_agg_large_num_groups('sum', 2) - 814±10ms 14.0±0.4ms 0.02 groupby.ResampleDask.time_agg_large_num_groups('sum', 2) - 760±8ms 12.0±0.3ms 0.02 groupby.ResampleDask.time_agg_large_num_groups('sum', 1) - 490±6ms 7.13±0.2ms 0.01 groupby.GroupByDask.time_agg_large_num_groups('sum', 1)

Earlier benchmarks:

This builds on #5950 so that should be reviewed and merged first.

!!!

The only failures are in test_units.py so now I think we can figure out how to implement this cleanly.

FAILED xarray/tests/test_units.py::TestDataArray::test_computation_objects[float64-method_groupby-data] FAILED xarray/tests/test_units.py::TestDataArray::test_computation_objects[float64-method_groupby_bins-data] FAILED xarray/tests/test_units.py::TestDataArray::test_computation_objects[int64-method_groupby-data] FAILED xarray/tests/test_units.py::TestDataArray::test_computation_objects[int64-method_groupby_bins-data] FAILED xarray/tests/test_units.py::TestDataArray::test_resample[float64] - pi... FAILED xarray/tests/test_units.py::TestDataArray::test_resample[int64] - pint... FAILED xarray/tests/test_units.py::TestDataset::test_computation_objects[float64-data-method_groupby_bins] FAILED xarray/tests/test_units.py::TestDataset::test_computation_objects[int64-data-method_groupby_bins] FAILED xarray/tests/test_units.py::TestDataset::test_resample[float64-data] FAILED xarray/tests/test_units.py::TestDataset::test_resample[int64-data] - p...

I like @max-sixty's suggestion of generating the reductions like generate_ops.py. It seems like a good first step would be to refactor the existing reductions in a separate PR.

OK CI isn't using the numpy_groupies code path for reasons I don't understand. Does anyone see a reason why this might happen?

Benchmarks are looking good (npg=True means use numpy groupies). Big gains (10-20x) for large number of groups (100), especially with dask.

``` [ 2.78%] ··· groupby.GroupBy.time_agg_large_num_groups ok [ 2.78%] ··· ======== ========== =========== ========== =========== -- ndim / npg
-------- --------------------------------------------- method 1 / True 1 / False 2 / True 2 / False ======== ========== =========== ========== =========== sum 8.38±0ms 101±0ms 9.54±0ms 136±0ms
mean 7.12±0ms 101±0ms 9.74±0ms 148±0ms
======== ========== =========== ========== ===========

[ 5.56%] ··· groupby.GroupBy.time_agg_small_num_groups ok [ 5.56%] ··· ======== ========== =========== ========== =========== -- ndim / npg
-------- --------------------------------------------- method 1 / True 1 / False 2 / True 2 / False ======== ========== =========== ========== =========== sum 8.27±0ms 4.55±0ms 9.07±0ms 8.46±0ms mean 7.19±0ms 4.50±0ms 9.24±0ms 8.36±0ms ======== ========== =========== ========== ===========

[ 8.33%] ··· groupby.GroupBy.time_init ok [ 8.33%] ··· ====== ========== ndim
------ ---------- 1 1.72±0ms 2 4.06±0ms ====== ==========

[ 11.11%] ··· groupby.GroupByDask.time_agg_large_num_groups ok [ 11.11%] ··· ======== ========== =========== ========== =========== -- ndim / npg
-------- --------------------------------------------- method 1 / True 1 / False 2 / True 2 / False ======== ========== =========== ========== =========== sum 8.41±0ms 202±0ms 9.93±0ms 226±0ms
mean 7.83±0ms 197±0ms 10.7±0ms 213±0ms
======== ========== =========== ========== ===========

[ 13.89%] ··· groupby.GroupByDask.time_agg_small_num_groups ok [ 13.89%] ··· ======== ========== =========== ========== =========== -- ndim / npg
-------- --------------------------------------------- method 1 / True 1 / False 2 / True 2 / False ======== ========== =========== ========== =========== sum 8.41±0ms 8.99±0ms 10.5±0ms 12.5±0ms mean 7.98±0ms 8.67±0ms 10.1±0ms 12.2±0ms ======== ========== =========== ========== ===========

[ 16.67%] ··· groupby.GroupByDask.time_init ok [ 16.67%] ··· ====== ========== ndim
------ ---------- 1 1.77±0ms 2 4.06±0ms ====== ==========

[ 36.11%] ··· groupby.Resample.time_agg_large_num_groups ok [ 36.11%] ··· ======== ========== =========== ========== =========== -- ndim / npg
-------- --------------------------------------------- method 1 / True 1 / False 2 / True 2 / False ======== ========== =========== ========== =========== sum 17.2±0ms 83.3±0ms 17.0±0ms 93.5±0ms mean 15.5±0ms 91.0±0ms 17.4±0ms 101±0ms
======== ========== =========== ========== ===========

[ 38.89%] ··· groupby.Resample.time_agg_small_num_groups ok [ 38.89%] ··· ======== ========== =========== ========== =========== -- ndim / npg
-------- --------------------------------------------- method 1 / True 1 / False 2 / True 2 / False ======== ========== =========== ========== =========== sum 16.7±0ms 12.3±0ms 16.7±0ms 13.3±0ms mean 15.2±0ms 12.5±0ms 19.3±0ms 13.9±0ms ======== ========== =========== ========== ===========

[ 41.67%] ··· groupby.Resample.time_init ok [ 41.67%] ··· ====== ========== ndim
------ ---------- 1 7.46±0ms 2 7.26±0ms ====== ==========

[ 44.44%] ··· groupby.ResampleDask.time_agg_large_num_groups ok [ 44.44%] ··· ======== ========== =========== ========== =========== -- ndim / npg
-------- --------------------------------------------- method 1 / True 1 / False 2 / True 2 / False ======== ========== =========== ========== =========== sum 22.3±0ms 561±0ms 28.3±0ms 607±0ms
mean 22.2±0ms 344±0ms 27.3±0ms 371±0ms
======== ========== =========== ========== ===========

[ 47.22%] ··· groupby.ResampleDask.time_agg_small_num_groups ok [ 47.22%] ··· ======== ========== =========== ========== =========== -- ndim / npg
-------- --------------------------------------------- method 1 / True 1 / False 2 / True 2 / False ======== ========== =========== ========== =========== sum 17.7±0ms 31.2±0ms 20.0±0ms 34.2±0ms mean 17.2±0ms 24.4±0ms 19.9±0ms 26.6±0ms ======== ========== =========== ========== ===========

[ 50.00%] ··· groupby.ResampleDask.time_init ok [ 50.00%] ··· ====== ========== ndim
------ ---------- 1 7.43±0ms 2 6.91±0ms == ```

Maybe it's also on the pint side? Even if numpy_groupies supports the like argument it will crash because pint doesn't support asanyarray.

cc @keewis

Maybe it's also on the pint side? Even if numpy_groupies supports the like argument it will crash because pint doesn't support asanyarray.

```python import numpy as np import pint import dask.array as da

pint crashes:

np.asanyarray([1, 2], like=pint.Quantity(1, "s")) Traceback (most recent call last):

File "<ipython-input-11-f8c20802ca08>", line 1, in <module> np.asanyarray([1, 2], like=pint.Quantity(1, "s"))

TypeError: no implementation found for 'numpy.asanyarray' on types that implement array_function: [<class 'pint.quantity.Quantity'>]

dask supports it:

np.asanyarray([1, 2], like=da.array(0)) Out[12]: dask.array<array, shape=(2,), dtype=int32, chunksize=(2,), chunktype=numpy.ndarray> ```

appears numpy_groupies is forcing the duck arrays to numpy arrays.

yes; this will require upstream changes

Looking at those unit errors it appears numpy_groupies is forcing the duck arrays to numpy arrays. Perhaps adding a like in the np.asanyarray() will do the trick? python C:\Miniconda\envs\xarray-tests\lib\site-packages\numpy_groupies\utils_numpy.py:199: in input_validation a = np.asanyarray(a) https://github.com/ml31415/numpy-groupies/blob/7a31d1f9bbd51111b4a4d01cf1df01a5b4827e85/numpy_groupies/utils_numpy.py#L199

We don't have any asv benchmarks for groupby currently. It would be good to add some!