issue_comments

A reproducible example would help but indexing with dask arrays is a bit limited.

On https://github.com/pydata/xarray/pull/5873 it's possible it will raise an error and ask you to compute the indexer. Also see https://github.com/dask/dask/issues/4156

EDIT: your slowdown is probably because it's compuing Sn multiple times. You could speed it up by calling compute once and passing a numpy array to isel

@dschwoerer are you sure that you are actually calculating the same thing in both cases? What exactly do the values of slc[d] look like? I would test thing on smaller inputs to verify. My guess is that you are inadvertently calculating something different, recalling that Xarray's broadcasting rules differ slightly from NumPy's.

I don't know much about indexing but that PR propagates a "new" indexes property as part of #1603 (work towards enabling more flexible indexing), it doesn't change anything about "indexing". I think the dask docs may be more relevant to what you may be asking about: https://docs.dask.org/en/latest/array-slicing.html

Yes, align checks index.equals(other) first, which has a shortcut for the same object.

The real mystery here is why time_filter.indexes['time'] and ds.indexes['time'] are not the same object. I guess this is likely due to lazy initialization of indexes, and should be fixed eventually by the explicit indexes refactor.

I think align tries to optimize that case, so maybe something's also possible there?

Can we short-circuit the special case where the index of the array used for slicing is the same object as the index being sliced, so no alignment is needed?

```python

time_filter.time._variable is ds.time._variable True %timeit xr.align(time_filter, ds.a) 477 ms ± 13.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) ``` the time spent on that align call could be zero!

Before #3319: ```

%timeit ds.a.values[time_filter] 158 ms ± 1.14 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%timeit ds.a.isel(time=time_filter.values) 2.57 s ± 3.65 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit ds.a.isel(time=time_filter) 3.12 s ± 37.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) After #3319: %timeit ds.a.values[time_filter] 158 ms ± 2.2 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%timeit ds.a.isel(time=time_filter.values) 665 ms ± 6.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit ds.a.isel(time=time_filter) 1.15 s ± 1.55 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) ``` Good job!

https://github.com/pydata/xarray/pull/3319 gives us about a 2x performance boost. It could likely be much faster, but at least this fixes the regression.

Yes, I'm seeing similar numbers, about 10x slower indexing in a DataArray. This seems to have gotten slower over time. It would be good to track this down and add a benchmark!

On master I'm seeing

``` %timeit ds.a.isel(time=time_filter) 3.65 s ± 29.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit ds.a.isel(time=time_filter.values) 2.99 s ± 15 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit ds.a.values[time_filter] 227 ms ± 6.59 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) ```

Can someone else reproduce?

In an effort to reduce the issue backlog, I'll close this, but please reopen if you disagree

@WeatherGod - are you reading data from netCDF files by chance?

If so, can you share the compression/chunk layout for those (ncdump -h -s file.nc)?

@WeatherGod does adding something like da = da.chunk({'time': 1}) reproduce this with your example?

@WeatherGod do you have a reproducible example? I'm happy to have a look

My measurements: ```

%timeit ds.a.isel(time=time_filter) 1 loop, best of 3: 906 ms per loop %timeit ds.a.isel(time=time_filter.values) 1 loop, best of 3: 447 ms per loop %timeit ds.a.values[time_filter] 10 loops, best of 3: 169 ms per loop ```

Given the size of this gap, I suspect this could be improved with some investigation and profiling, but there is certainly an upper-limit on the possible performance gain.

One simple example is that indexing the dataset needs to index both 'a' and 'time', so it's going to be at least twice as slow as only indexing 'a'. So the second indexing expression ds.a.isel(time=time_filter.values) is only 447/(169*2) = 1.32 times slower than the best case scenario.

Another part of the matrix of possibilities. Takes about half the time if you pass time_filter.values (numpy array) rather than the time_filter DataArray: python %timeit ds.a.isel(time=time_filter.values) 1.3 s ± 67.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

@rabernat that's a good solution where it's a slice

When is a time that it needs to align a bool array? If you try and pass an array of unequal length, it doesn't work anyway:

```python In [12]: ds.a.isel(time=time_filter[:-1])

IndexError: Boolean array size 54999999 is used to index array with shape (55000000,). ```

I don't have experience using isel with boolean indexing. (Although the docs on positional indexing claim it is supported.) My guess is that that the time is being spent aligning the indexer with the array, which is unnecessary since you know they are already aligned. Probably not the most efficient pattern for xarray.

Here's how I would recommend writing the query using label-based selection: python %timeit ds.a.sel(time=slice(50_001, None)) 117 ms ± 5.29 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)