issue_comments

Hey, thanks @dcherian for taking over and merging this PR ! (and sorry for not being active on it myself for the past year...)

@mathause This PR goes beyond what is currently implemented in numpy. For now, all weighted quantiles PR on numpy are more or less based on "linear" method (method 7) and none have been merged. I plan to work on integrating weights with the other interpolation methods but don't have the time right now. I'll probably pick some ideas from here.

As for the numerics here, everything looks good to me. The only limitations I can see are: - This only handles sampling weights, which is fine I guess. - Some interpolation methods are missing, they can be added later. - ~A nan_weighted_quantile could also be interesting to add~

FYI, weighted quantiles topic will be discussed in numpy's triage meeting of today (17:00 UTC). I'm not a maintainer but I'm sure you are welcomed to join in if you are interested. Meeting information: https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg

list type is only used on this handler. Its usage is never documented and it's not even part of dask rechunk api either. It would make sense to deprecate it I guess.

I also noticed the Tuples types Tuple[int, ...] and Tuple[Tuple[int, ...], ...], are accepted and handled at DataArray level but are not at Dataset level. The handler is: if isinstance(chunks, (tuple, list)): chunks = dict(zip(self.dims, chunks))

Would it make sense to move this in Dataset to have the same api for both ?

I'll drop a PR, it might be easier to try and play with this than a piece of code lost in an issue.

Okay I could re do my test. If I manually call compute() before doing isel(......) my whole computation takes about 5.65 seconds. However if I try with my naive patch it takes 32.34 seconds.

I'm sorry I cannot share as is my code, the relevant portion is really in the middle of many things. I'll try to get a minimalist version of it to share with you.

@pl-marasco Ok that's strange. I should have saved my use case :/ I will try to reproduce it and will provide a gist of it soon.

@pl-marasco Thanks for the example ! With it I have the same result as you, it takes the same time with patch or with compute.

However, I could construct an example giving very different results. It is quite close to my original code:

``` time_start = time.perf_counter() COORDS = dict( time=pd.date_range("2042-01-01", periods=200, freq=pd.DateOffset(days=1)), ) da = xr.DataArray( np.random.rand(200 * 3500 * 350).reshape((200, 3500, 350)), dims=('time', 'x', 'y'), coords=COORDS ).chunk(dict(time=-1, x=100, y=100))

resampled = da.resample(time="MS")

for label, sample in resampled:
    # sample = sample.compute()
    idx = sample.argmax('time')
    sample.isel(time=idx)

time_elapsed = time.perf_counter() - time_start
print(time_elapsed, " secs")

``` (Basically I want for each month the first event occurring in it).

Without the patch and uncommenting sample = sample.compute(), it takes 5.7 secs. With the patch it takes 53.9 seconds.

I wrote a very naive fix, it works but seems to perform really slowly, I would appreciate some feedback (I'm a beginner with Dask). Basically, I added k = dask.array.asarray(k, dtype=np.int64) to do the exact same thing as with numpy. I can create a PR if it's better to review this

The patch: ``` class VectorizedIndexer(ExplicitIndexer): """Tuple for vectorized indexing.

All elements should be slice or N-dimensional np.ndarray objects with an
integer dtype and the same number of dimensions. Indexing follows proposed
rules for np.ndarray.vindex, which matches NumPy's advanced indexing rules
(including broadcasting) except sliced axes are always moved to the end:
https://github.com/numpy/numpy/pull/6256
"""

__slots__ = ()

def __init__(self, key):
    if not isinstance(key, tuple):
        raise TypeError(f"key must be a tuple: {key!r}")

    new_key = []
    ndim = None
    for k in key:
        if isinstance(k, slice):
            k = as_integer_slice(k)
        elif isinstance(k, np.ndarray) or isinstance(k, dask.array.Array):
            if not np.issubdtype(k.dtype, np.integer):
                raise TypeError(
                    f"invalid indexer array, does not have integer dtype: {k!r}"
                )
            if ndim is None:
                ndim = k.ndim
            elif ndim != k.ndim:
                ndims = [k.ndim for k in key if isinstance(k, np.ndarray)]
                raise ValueError(
                    "invalid indexer key: ndarray arguments "
                    f"have different numbers of dimensions: {ndims}"
                )
            if isinstance(k, dask.array.Array):
                k = dask.array.asarray(k, dtype=np.int64)
            else:
                k = np.asarray(k, dtype=np.int64)
        else:
            raise TypeError(
                f"unexpected indexer type for {type(self).__name__}: {k!r}"
            )
        new_key.append(k)

    super().__init__(new_key)

```