issue_comments

Hello,

I'm not sure performances problematics were fully addressed (we're now forced to fully compute/load the selection expression) but changes made in the last versions makes this issue irrelevant and I think we can close it.

Thank you!

The .variable computation is fast but it cannot be directly used like you suggest: ``` dsx.where(sel.variable, drop=True)

TypeError: cond argument is <xarray.Variable (num_lines: 5761870, num_pixels: 71)> ... but must be a <class 'xarray.core.dataset.Dataset'> or <class 'xarray.core.dataarray.DataArray'> ```

Doing it like this seems to be working correctly (and is fast enough): dsx["x"]= sel.variable.compute() dsx.where(dsx["x"], drop=True)

_nadir variables have the same chunks and are way faster to read than the other ones (lot smaller).

Yeah that was another change I guess. We could extract out the variable using .variable.

.where(sel2.variable.compute(), drop=True)

do your "_nadir" variables have smaller chunk sizes or are slower to read for some reason?

sel = (dsx["longitude"] > 0) & (dsx["longitude"] < 100) sel.compute() This "compute" finishes and takes more than 80sec on both versions with a huge memory consumption (it loads the 4 coordinates and the result itself).

I know xarray has to keep more information regarding coordinates and dimensions but doing this (just dask arrays) : sel2 = (dsx["longitude"].data > 0) & (dsx["longitude"].data < 100) sel2.compute() Takes less than 6 seconds.

Does sel.compute() not finish?

Just tried it and it does not seem identical at all to what was happening earlier.

This is the kind of dataset I'm working

With this selection: sel = (dsx["longitude"] > 0) & (dsx["longitude"] < 100)

Old xarray takes a little less that 1 minute and less than 6GB of memory. New xarray with compute did not finish and had to be stopped before consuming my 16GB of memory.

The old code had: nonzeros = zip(clipcond.dims, np.nonzero(clipcond.values))

This loaded the array once and then passed numpy values to the indexing code.

Now, the dask array is passed to the indexing code and is computed many times . #5873 raises an error saying boolean indexing with dask arrays is not allowed.

For here just do ds.where(sel.compute(), drop=True). It's identical to what was happening earlier.

I think we should close this.

I am a bit puzzled here... The dask graph looks identical, so it must be the way the indexers are constructed.

The major difference I can find is: The old version used np.unique while the new version uses xarrays cond.any(..)

Maybe someone with more experience in dask can help out?

Git bisect pinpoints this to https://github.com/pydata/xarray/pull/6690 which funny enough, is my PR haha. I will look into it when I find time :)

Can confirm, on my machine it went from 520ms to 5s