issue_comments

I think reindex would need to be changed

that's true, I only tried the special case where the data that would be used to do the forward fill is included in the result.

I guess this works but it's a bit cumbersome

yeah, to_dataset is probably not the right tool for pointwise indexing.

it does not fail if one of the sensors in the query list is missing

if I understand correctly, you would like to index with arbitrary values for time, but would like an error for missing values of sensor. Unfortunately, I don't think that is possible using a single call to sel. Instead, you could set the fill_value parameter of reindex to some other value (for example, -np.inf) and then drop these values after the pointwise indexing.

Thanks for the response. I think reindex would need to be changed as well because this code:

python sensor_data.reindex({ 'time': [1], 'sensor': ['A', 'B'] }, method='ffill')

Is not equivalent to this code: python sensor_data.reindex({ 'time': [1], 'sensor': ['A', 'B'] }).ffill(dim='time').ffill(dim='sensor')

So if I understand your to_dataset idea correctly, you are proposing:

python ds = sensor_data.to_dataset(dim='sensor') xr.concat([ ds[sensor].sel({'time': time}, method='ffill', drop=True) for sensor, time in zip(['A', 'A', 'A', 'B', 'C'], [0, 1, 2, 0, 0]) ], dim='sample')

I guess this works but it's a bit cumbersome and unlikely to be fast. I think there must be something I'm not understanding here - I'm not familiar with all the nuances of the xarray api.

Your idea of reindex followed by sel is an interesting one, but it does do something slightly different than what I was asking for: it does not fail if one of the sensors in the query list is missing, but rather inserts a NaN. I suppose you could fix this by doing an extra check afterwards, assuming that your original pre-reindex data contained no NaNs.

In general min(S*N,T*N) could be much larger than S*T, so for big queries it's quite possible that you wouldn't have enough space to allocate the intermediate even if you could fit 100s of copies of the original S*T matrix. Using a dask cluster would make this situation less likely of course, but it seems like it would be better to avoid all this copying (even on a beefy cluster) even if just for performance reasons.

reindex does not have to be changed since we can just call e.g. ffill with the dim parameter for this to work: python arr.reindex(...).ffill(dim="dim")

This really depends on how you intend to use the result of the indexing. For example, if you don't really need the big matrix, you could just convert the DataArray to a Dataset where the sensor dimension is the names of the variables (using to_dataset(dim="sensor"), or construct it that way). If you do need the matrix, this might be slightly better (you still end up allocating a T * (S + n) array): python arr.reindex(sensor=["A", "B", "C"]).sel({"sensor": ..., "time": ...}, method="ffill") but if you really care about the memory allocated at once, you might be better off using dask: python arr.chunk({"time": 100}).reindex(...).sel(...)

If all of that is not an option, I guess we might be able add a method_kwargs parameter (not sure if there is a better option, though).

Thanks for the suggestion. One issue with this alternative is it creates a potentially large intermediate object.

If you have T times and S sensors, and want to sample them at N (time, sensor) pairs, then the intermediate object with your approach has size T*N (if you index sensors first) or S*N (if you index time first). If you can index both dimensions in one sel call then we should only need to allocate memory for the result of size N, which is considerably better.

Could you split it in two calls or does this not do what you want?

python sensor_data.sel({ 'sensor': xr.DataArray(['A', 'A', 'A', 'B', 'C'], dims=['sample'])}).sel({ 'time': xr.DataArray([0, 1, 2, 0, 0], dims=['sample']) }, method='ffill')

I guess it would also make sense to have this in reindex if you did decide to add it.