What happened?

Hi all, as the title already summarizes, I'm running into performance issues when aggregating over the time-dimension of a 3D DataArray using the quantiles method with skipna=True. See the section below for some dummy data that represents what I'm working with (e.g., similar to this). Aggregating over the time-dimension of this dummy data I'm getting the following wall times:

| | | | | --------------- | --------------- | --------------- | | 1 | da.median(dim='time', skipna=True) | 1.35 s | | 2 | da.quantile(0.95, dim='time', skipna=False) | 5.95 s | | 3 | da.quantile(0.95, dim='time', skipna=True) | 6 min 6s |

I'm currently using a compute node with 40 CPUs and 180 GB RAM. Here is what the resource utilization looks like. First small bump are 1 and 2. Second longer peak is 3.

In this small example, the process at least finishes after a few seconds. With my actual dataset the quantile calculation takes hours...

I guess the following issue is relevant and should be revived: https://github.com/numpy/numpy/issues/16575

Are there any possible work-arounds?

What did you expect to happen?

No response

Minimal Complete Verifiable Example

```Python import pandas as pd import numpy as np import xarray as xr

Create dummy data with 20% random NaNs

size_spatial = 2000 size_temporal = 20 n_nan = int(size_spatial*20.2)

time = pd.date_range("2000-01-01", periods=size_temporal) lat = np.random.uniform(low=-90, high=90, size=size_spatial) lon = np.random.uniform(low=-180, high=180, size=size_spatial) data = np.random.rand(size_temporal, size_spatial, size_spatial) index_nan = np.random.choice(data.size, n_nan, replace=False) data.ravel()[index_nan] = np.nan

Create DataArray

da = xr.DataArray(data=data, dims=['time', 'x', 'y'], coords={'time': time, 'x': lon, 'y': lat}, attrs={'nodata': np.nan})

Calculate 95th quantile over time-dimension

da.quantile(0.95, dim='time', skipna=True) ```

MVCE confirmation

• [x] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [x] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [x] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

What happened?

When performing groupby-/resample-reduce operations (e.g., ds.resample(time="6h").mean() as shown here) the alignment of chunks is not preserved when flox is disabled:

...whereas the alignment is preserved when flox is enabled:

What did you expect to happen?

The alignment of chunks is preserved whether using flox or not.

Minimal Complete Verifiable Example

```Python import pandas as pd import numpy as np import xarray as xr

size_spatial = 1000 size_temporal = 200 time = pd.date_range("2000-01-01", periods=size_temporal, freq='h') lat = np.random.uniform(low=-90, high=90, size=size_spatial) lon = np.random.uniform(low=-180, high=180, size=size_spatial) data = np.random.rand(size_temporal, size_spatial, size_spatial)

da = xr.DataArray(data=data, dims=['time', 'x', 'y'], coords={'time': time, 'x': lon, 'y': lat}).chunk({'time': -1, 'x': 'auto', 'y': 'auto'})

Chunk alignment not preserved

with xr.set_options(use_flox=False): da_1 = da.copy(deep=True) da_1 = da_1.resample(time="6h").mean()

Chunk alignment preserved

with xr.set_options(use_flox=True): da_2 = da.copy(deep=True) da_2 = da_2.resample(time="6h").mean() ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Is your feature request related to a problem?

With satellite remote sensing data, you sometimes end up with a blown up DataArray/Dataset because individual acquisitions have been saved in slices:

One could then aggregate these slices with something like this:

python ds.coords['time'] = ds.time.dt.floor('1H') # or .ceil ds = ds_copy.groupby('time').mean()

However, this would miss cases where one slice has been acquired before and the other after a specific hour. The pandas.DatetimeIndex.snap method could be a good alternative for such cases.

Describe the solution you'd like

In addition to the floor, ceil and round methods, it would be great to also implement pandas.DatetimeIndex.snap.

Describe alternatives you've considered

No response

Additional context

No response

What is your issue?

I noticed that the docstrings of the aggregation methods are mostly written in the same style, e.g.: "Reduce this Dataset's data by applying xy along some dimension(s).". Let's say a user is interested in calculating the variance and searches for the appropriate method. Neither xarray.DataArray.var nor xarray.Dataset.var will be returned (see here), because "variance" is not mentioned at all in the docstrings. Same problem exists for other methods like .std, .prod, .cumsum, .cumprod, and probably others.

https://github.com/pydata/xarray/issues/6793 is related, but I guess it already has enough tasks.