issue_comments

here's the dask issue about np.percentile's missing axis parameter: dask/dask#2824

yes. you'll need to have 1 chunk along the dimension being reduced (here 'time').

The reason is that there is no approximate quantile algorithm implemented in dask. So this implementation is just calling np.nanpercentile on each block of the array. If you're reducing along time, each block must have the entire time axis for this to work.

keewis' quantile function works fine on 1 year of data (as in the MVCE code sample) but does not like working on 10 years of data:

ValueError: dimension 'time' on 0th function argument to apply_ufunc with dask='parallelized' consists of multiple chunks, but is also a core dimension. To fix, rechunk into a single dask array chunk along this dimension, i.e., ``.chunk({'time': -1})``, but beware that this may significantly increase memory usage.

that would work if we would be working with variables, but Dataset.quantile does a lot more that I didn't want to copy.

We should issue a release soon though.

I agree

seems to make quantile work, but it might be a bad idea to monkeypatch. T

Personally I would add this quantile function and then do quantile(ds, q=0.99, dim="SOMETHING"). We should issue a release soon though.

Thanks a lot for this!

sorry, my mistake. I've been testing on master which has #3559 merged and thus works with dask arrays. Unless someone has a better idea, I'd suggest copying that approach for now: using apply_ufunc to apply np.nanpercentile. ~~Ideally, that would change the quantile call to~~ python percentiles = quantile(ds, q=percentile / 100, dim="time") ~~I'll try to write a working quantile function.~~ Edit: see below, with that you should be able to use python ds.quantile(...)

This ```python def quantile(self, q, dim=None, interpolation="linear", keep_attrs=None): from xarray.core.computation import apply_ufunc from xarray.core import utils

scalar = utils.is_scalar(q) 
q = np.atleast_1d(np.asarray(q, dtype=np.float64)) 
if dim is None: 
    dim = self.dims

if utils.is_scalar(dim): 
    dim = [dim]

def _wrapper(npa, **kwargs): 
    # move quantile axis to end. required for apply_ufunc 
    return np.moveaxis(np.nanpercentile(npa, **kwargs), 0, -1)

axis = np.arange(-1, -1 * len(dim) - 1, -1) 
result = apply_ufunc( 
    _wrapper, 
    self, 
    input_core_dims=[dim], 
    exclude_dims=set(dim), 
    output_core_dims=[["quantile"]], 
    output_dtypes=[np.float64], 
    output_sizes={"quantile": len(q)}, 
    dask="parallelized", 
    kwargs={"q": q * 100, "axis": axis, "interpolation": interpolation}, 
)

# for backward compatibility 
result = result.transpose("quantile", ...) 
if scalar: 
    result = result.squeeze("quantile") 
if keep_attrs: 
    result.attrs = self._attrs

return result

xr.Variable.quantile = quantile `` seems to makequantile` work, but it might be a bad idea to monkeypatch. Thoughts, @dcherian?

Seems like I do need it:

python percentiles = ds.quantile(q=percentile / 100, dim="time") returns

TypeError: quantile does not work for arrays stored as dask arrays. Load the data via .compute() or .load() prior to calling this method.

you should be able to just use quantile. I called compute to demonstrate it returns the same values as np.percentile, but you don't need it with quantile.

Thanks for that but ideally I wouldn't use compute(). The code worked using a previous version of dask (0.16.1) so I'm hoping there might be an easy fix for dask 2.9.0...

the reason for this is that the percentile implementation of dask arrays does not support axis. I think you can achieve the same thing by either using compute() (which is probably not desirable) or by using quantile: python In [13]: # Get the percentiles at each lat/lon: ...: ds = ds.compute() ...: percentiles = ds.reduce(np.percentile, q=percentile, dim='time') ...: quantiles = ds.quantile(q=percentile / 100, dim="time") ...: xr.testing.assert_identical(percentiles.drop_vars("height"), quantiles.drop_vars("quantile")) the only difference is that the unused coordinate height disappears and a quantile dimension is added.