issues

Ported from #1724, comment by @shoyer

In the long term, it would be nice to get ride of these uses of _data, maybe by switching entirely from our lazy array classes to Dask.

The subtleties of checking _data vs data are undesirable, e.g., consider the bug on these lines: https://github.com/pydata/xarray/blob/1a012080e0910f3295d0fc26806ae18885f56751/xarray/core/formatting.py#L212-L213

MCVE Code Sample

I want to minimize the number of calls .item() within my data analysis. It often happens

1. when putting a 0d-DataArray into a slice python da = xr.DataArray([0.5, 4.5, 2.5], dims=['x'], coords={'x': [0, 1, 2]}) da[: da.argmax()] -> TypeError: 'DataArray' object cannot be interpreted as an integer

2. when using a 0d-DataArray for selecting python da = xr.DataArray([0.5, 4.5, 2.5], dims=['x'], coords={'x': [0, 0, 2]}) da.sel(x=da['x'][0]) -> IndexError: arrays used as indices must be of integer (or boolean) type

Both cases, I need to call '.item()'. It is not a big issue, but I think it would be nice if xarray becomes more self-contained.

In #4219 we added ndrolling. However, nanreduce, such as ds.rolling(x=3, y=2).mean() calls np.nanmean which copies the strided-array into a full-array. This is memory-inefficient.

We can implement inhouse-nanreduce methods for the strided array. For example, our .nansum currently does make a strided array -> copy the array -> replace nan by 0 -> sum but we can do instead replace nan by 0 -> make a strided array -> sum This is much more memory efficient.

• [x] Closes #2007, #2011
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Added some boundary options for rolling.construct. Currently, the option names are inherited from np.pad, ['edge' | 'reflect' | 'symmetric' | 'wrap']. Do we want a more intuitive name, such as periodic?

np.nonzero to DataArray returns a wrong result,

python In [4]: da = xr.DataArray(np.arange(12).reshape(4, 3), dims=['x', 'y'], ...: coords={'x': [0, 1, 2, 3], 'y': ['a', 'b', 'c']}) ...: np.nonzero(da) ...: Out[4]: <xarray.DataArray (x: 2, y: 11)> array([[0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3], [1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2]]) Coordinates: * x (x) int64 0 1 2 3 * y (y) <U1 'a' 'b' 'c'

Problem description

Apparently, the dimensions and the coordinates conflict each other. I think we can have our own nonzero method, which may return a Dataset consisting of indexes and appropriate coordinates.

Output of xr.show_versions()

groupby works inconsistently depending on the dimension order of a DataArray. Furthermore, in some cases, this causes a corrupted object.

python In [4]: data = xr.DataArray( ...: np.random.randn(4, 2), ...: dims=['x', 'z'], ...: coords={'x': ['a', 'b', 'a', 'c'], 'y': ('x', [0, 1, 0, 2])} ...: ) ...: ...: data.groupby('x').mean() Out[4]: <xarray.DataArray (x: 3, z: 2)> array([[ 0.95447186, -1.14467028], [ 0.76294958, 0.3751244 ], [-0.41030223, -1.35344548]]) Coordinates: * x (x) object 'a' 'b' 'c' Dimensions without coordinates: z groupby works fine (although this drops nondimensional coordinate y, related to #3745).

However, groupby does not give a correct result if we work on the second dimension, python In [5]: data.T.groupby('x').mean() # <--- change the dimension order, and do the same thing Out[5]: <xarray.DataArray (z: 2, x: 3)> array([[ 0.95447186, 0.76294958, -0.41030223], [-1.14467028, 0.3751244 , -1.35344548]]) Coordinates: * x (x) object 'a' 'b' 'c' y (x) int64 0 1 0 2 # <-- the size must be 3!! Dimensions without coordinates: z

The bug has been discussed in #2944 and solved, but I found this is still there.

We have a bunch of terms to describe the xarray structure, such as dimension, coordinate, dimension coordinate, etc.. Although it has been discussed in #1295 and we tried to use the consistent terminology in our docs, it looks still not easy for users to understand our functionalities.

In #2399, @horta wrote a list of definitions (https://drive.google.com/file/d/1uJ_U6nedkNe916SMViuVKlkGwPX-mGK7/view?usp=sharing). I think it would be nice to have something like this in our docs.

Any thought?

From a discussion in #1473 comment

It would be convenient if we have multi-dimensional reindex method, where we consider dimensions and coordinates of indexers. The proposed outline by @shoyer is

• Given reindex arguments of the form dim=array where array is a 1D unlabeled array/list, convert them into DataArray(array, [(dim, array)]).
• Do multi-dimensional indexing with broadcasting like sel, but fill in NaN for missing values (we could allow for customizing this with a fill_value argument).
• Join coordinates like for sel, but coordinates from the indexers take precedence over coordinates from the object being indexed.

https://github.com/pydata/xarray/blob/5e801894886b2060efa8b28798780a91561a29fd/xarray/core/dataarray.py#L482-L489

In #1746, we added a validation in xr.DataArray.__setitem__ whether the coordinates consistency of array, key, and values are checked. In the current implementation, we call xr.DataArray.__getitem__ to use the existing coordinate validation logic, but it does unnecessary indexing and it may decrease the __setitem__ performance if the arrray is multidimensional.

We may need to optimize the logic here.

Is it reasonable to constantly monitor the performance of basic operations, such as Dataset construction, alignment, indexing, and assignment? (or are these operations too light to make a performance monitor?)

cc @jhamman @shoyer