Ported from #1724, comment by @shoyer

In the long term, it would be nice to get ride of these uses of _data, maybe by switching entirely from our lazy array classes to Dask.

The subtleties of checking _data vs data are undesirable, e.g., consider the bug on these lines: https://github.com/pydata/xarray/blob/1a012080e0910f3295d0fc26806ae18885f56751/xarray/core/formatting.py#L212-L213

MCVE Code Sample

I want to minimize the number of calls .item() within my data analysis. It often happens

1. when putting a 0d-DataArray into a slice python da = xr.DataArray([0.5, 4.5, 2.5], dims=['x'], coords={'x': [0, 1, 2]}) da[: da.argmax()] -> TypeError: 'DataArray' object cannot be interpreted as an integer

2. when using a 0d-DataArray for selecting python da = xr.DataArray([0.5, 4.5, 2.5], dims=['x'], coords={'x': [0, 0, 2]}) da.sel(x=da['x'][0]) -> IndexError: arrays used as indices must be of integer (or boolean) type

Both cases, I need to call '.item()'. It is not a big issue, but I think it would be nice if xarray becomes more self-contained.

In #4219 we added ndrolling. However, nanreduce, such as ds.rolling(x=3, y=2).mean() calls np.nanmean which copies the strided-array into a full-array. This is memory-inefficient.

We can implement inhouse-nanreduce methods for the strided array. For example, our .nansum currently does make a strided array -> copy the array -> replace nan by 0 -> sum but we can do instead replace nan by 0 -> make a strided array -> sum This is much more memory efficient.

I think we can continue the discussion we have in #1426 about MultiIndex here.

In comment , @shoyer recommended to remove MultiIndex from public API.

I agree with this, as long as my codes work with this improvement.

I think if we could have a list of possible MultiIndex use cases here, it would be easier to deeply discuss and arrive at a consensus of the future API.

Current limitations of MultiIndex are + It drops scalar coordinate after selection #1408, #1491 + It does not support to serialize to NetCDF #1077 + Stack/unstack behaviors are inconsistent #1431

I am sorry but I unintentionally pushed my working scripts to xarray.master. (I thought it is not allowed and I was not careful.)

What is the best way to reset this? I'm thinking to do in my local, and force push again, but I'm afraid that I do another wrong thing...

I apologize for my mistake.

np.nonzero to DataArray returns a wrong result,

python In [4]: da = xr.DataArray(np.arange(12).reshape(4, 3), dims=['x', 'y'], ...: coords={'x': [0, 1, 2, 3], 'y': ['a', 'b', 'c']}) ...: np.nonzero(da) ...: Out[4]: <xarray.DataArray (x: 2, y: 11)> array([[0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3], [1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2]]) Coordinates: * x (x) int64 0 1 2 3 * y (y) <U1 'a' 'b' 'c'

Problem description

Apparently, the dimensions and the coordinates conflict each other. I think we can have our own nonzero method, which may return a Dataset consisting of indexes and appropriate coordinates.

Output of xr.show_versions()

groupby works inconsistently depending on the dimension order of a DataArray. Furthermore, in some cases, this causes a corrupted object.

python In [4]: data = xr.DataArray( ...: np.random.randn(4, 2), ...: dims=['x', 'z'], ...: coords={'x': ['a', 'b', 'a', 'c'], 'y': ('x', [0, 1, 0, 2])} ...: ) ...: ...: data.groupby('x').mean() Out[4]: <xarray.DataArray (x: 3, z: 2)> array([[ 0.95447186, -1.14467028], [ 0.76294958, 0.3751244 ], [-0.41030223, -1.35344548]]) Coordinates: * x (x) object 'a' 'b' 'c' Dimensions without coordinates: z groupby works fine (although this drops nondimensional coordinate y, related to #3745).

However, groupby does not give a correct result if we work on the second dimension, python In [5]: data.T.groupby('x').mean() # <--- change the dimension order, and do the same thing Out[5]: <xarray.DataArray (z: 2, x: 3)> array([[ 0.95447186, 0.76294958, -0.41030223], [-1.14467028, 0.3751244 , -1.35344548]]) Coordinates: * x (x) object 'a' 'b' 'c' y (x) int64 0 1 0 2 # <-- the size must be 3!! Dimensions without coordinates: z

The bug has been discussed in #2944 and solved, but I found this is still there.

.sel method usually keeps selected coordinate value as a scalar coordinate

```python In[4] ds1 = xr.Dataset({'foo': (('x',), [1, 2, 3])}, {'x': [1, 2, 3], 'y': 'a'})

Out[4]: <xarray.Dataset> Dimensions: () Coordinates: y <U1 'a' x int64 1 Data variables: foo int64 1 ```

But in MultiIndex case, does not.

python In[5] ds2 = xr.Dataset({'foo': (('x',), [4, 5, 6])}, {'x': [1, 2, 3], 'y': 'b'}) ds = xr.concat([ds1, ds2], dim='y').stack(yx=['y', 'x']) ds.sel(x=2) Out[5] <xarray.Dataset> Dimensions: (y: 2) Coordinates: * y (y) object 'a' 'b' Data variables: foo (y) int64 2 5 x is gone. Is it a desired behavior?

We have a bunch of terms to describe the xarray structure, such as dimension, coordinate, dimension coordinate, etc.. Although it has been discussed in #1295 and we tried to use the consistent terminology in our docs, it looks still not easy for users to understand our functionalities.

In #2399, @horta wrote a list of definitions (https://drive.google.com/file/d/1uJ_U6nedkNe916SMViuVKlkGwPX-mGK7/view?usp=sharing). I think it would be nice to have something like this in our docs.

Any thought?

Code Sample, a copy-pastable example if possible

```python

import numpy as np import xarray as xr

da = xr.DataArray(np.arange(12).reshape(3, 4), dims=['x', 'y'], ... coords={'x': [0, 1, 1], 'x2': ('x', ['a', 'b', 'c'])}) grouped = da.groupby('x').mean('x') grouped <xarray.DataArray (x: 2, y: 4)> array([[0., 1., 2., 3.], [6., 7., 8., 9.]]) Coordinates: * x (x) int64 0 1 x2 (x) <U1 'a' 'b' 'c' # <-- this has length 3! Dimensions without coordinates: y ```

Problem description

groupby operation ignores the non-dimensional coordinate.

Expected Output

python <xarray.DataArray (x: 2, y: 4)> array([[0., 1., 2., 3.], [6., 7., 8., 9.]]) Coordinates: * x (x) int64 0 1 Dimensions without coordinates: y Compute mean of the coordinate. If not possible, drop it.

From a discussion in #1473 comment

It would be convenient if we have multi-dimensional reindex method, where we consider dimensions and coordinates of indexers. The proposed outline by @shoyer is

• Given reindex arguments of the form dim=array where array is a 1D unlabeled array/list, convert them into DataArray(array, [(dim, array)]).
• Do multi-dimensional indexing with broadcasting like sel, but fill in NaN for missing values (we could allow for customizing this with a fill_value argument).
• Join coordinates like for sel, but coordinates from the indexers take precedence over coordinates from the object being indexed.

Currently I'm using xarray to handle multiple images (typically, a sequence of images), and I feel it would be convenient if xarray supports image related functions.

There may be many possibilities, but particular methods I want to have in xarray are 1. xr.open_image(File)
Currently, I open image by PILLOW, convert to np.ndarray, extract its attributes, then construct xr.DataArray from them. If I can do it by 1 line, it would be very great.

1. xr.DataArray.expand_dims(dim) I want to add additional channel dimension to grey scale images (size [W x H] -> [W x H x 1]), in order to pass them into convolutional neural network, which usually accepts 4-dimensional tensor [Batch x W x H x channel].

Image (possibly also video?) is naturally high-dimensional and I guess it would fit xarray's concept. Is this sufficiently broad interest?

I noticed that some users experiencing troubles to restore xarray objects that is created xarray < 0.8.

Is there any possibility to add a function to support unpickling old object, such as xr.unpickle_legacy(file, version)? We previously recommended using pickle to store xarray objects (at least for the short term use, maybe).

xref (private repo) gafusion/OMFIT-source#2652

Currently, we disallow to save complex values to a netCDF file. Maybe netCDF itself does not support complex values, but there may be some workarounds. It would be very handy for me.

The most naive workaround may be to split each complex value into a real and imaginary part, add some flags, and restore it when loading them from the file. Maybe we may add a special suffix to the variable name? ```python

ds = xr.Dataset({'a': ('x': [1+2j, 2+3j])}, coords={'x': [0, 1]}) ds.to_netcdf('tmp.nc', encode_complex=True) xr.load_netcdf('tmp.nc') <xarray.Dataset> Dimensions: (x: 2) Coordinates: * x (x) int64 0 1 Data variables: a__real__ (x) int64 1 2 a__imag__ (x) int64 2 3 xr.load_netcdf('tmp.nc', decode_complex=True) <xarray.Dataset> Dimensions: (x: 2) Coordinates: * x (x) int64 0 1 Data variables: a (x) complex128 (1+2j) (2+3j) ```

I think there may be a better way. Any thoughts are welcome :)

p.s. I just found that engine=h5netcdf can save complex values, but the file becomes an invalid netcdf file. I'm not sure if it worth the trouble just to make a valid netCDF file.

If using xarray inside VScode with darkmode, the new html repr has a small contrast of the text color and background.

Maybe the text color comes from the default setting, but the background color is not. In light mode, it looks nice.

Versions

Related to #138

Why is keep_attrs option in reduce method set to FALSE by default? I feel it is more natural to keep all the attrs after some computation that returns xr.DaraArray Such as data*100. (By it is not possible to set this option TRUE when using an operator.) Is it an option to move this option to init method, in case of TRUE all the attrs are tracked after computations of the object and also the object generated from this object?

MCVE Code Sample

```python da = xr.DataArray([0, 1], dims=['x'], coords={'x': [0, 1], 'y': 'a'}) db = xr.DataArray([2, 3], dims=['x'], coords={'x': [0, 1], 'y': 'b'}) data = xr.concat([da, db], dim='x').set_index(xy=['x', 'y']) data.sel(y='a')

<xarray.DataArray (x: 4)> array([0, 1, 2, 3]) Coordinates: * x (x) int64 0 1 ```

Expected Output

```python

<xarray.DataArray (x: 2)> array([0, 1]) Coordinates: * x (x) int64 0 1 ```

Problem Description

Should select the array

Output of xr.show_versions()

Sorry for being quiet for a long time. I hope I could send a fix for this in a few days...

Tests are failing, which is caused by new release of dask==0.18.2. xref: dask/dask#3822

https://github.com/pydata/xarray/blob/5e801894886b2060efa8b28798780a91561a29fd/xarray/core/dataarray.py#L482-L489

In #1746, we added a validation in xr.DataArray.__setitem__ whether the coordinates consistency of array, key, and values are checked. In the current implementation, we call xr.DataArray.__getitem__ to use the existing coordinate validation logic, but it does unnecessary indexing and it may decrease the __setitem__ performance if the arrray is multidimensional.

We may need to optimize the logic here.

Is it reasonable to constantly monitor the performance of basic operations, such as Dataset construction, alignment, indexing, and assignment? (or are these operations too light to make a performance monitor?)

cc @jhamman @shoyer

From #2356, I noticed resample and groupby works nice for Dataset but not for DataArray

Code Sample, a copy-pastable example if possible

```python In [14]: import numpy as np ...: import xarray as xr ...: import pandas as pd ...: ...: time = pd.date_range('2000-01-01', freq='6H', periods=365 * 4) ...: ds = xr.Dataset({'foo': (('time', 'x'), np.random.randn(365 * 4, 5)), 'time': time, ...: 'x': np.arange(5)})

In [15]: ds Out[15]: <xarray.Dataset> Dimensions: (time: 1460, x: 5) Coordinates: * time (time) datetime64[ns] 2000-01-01 ... 2000-12-30T18:00:00 * x (x) int64 0 1 2 3 4 Data variables: foo (time, x) float64 -0.6916 -1.247 0.5376 ... -0.2197 -0.8479 -0.6719 ``ds.resample(time='M').mean()['foo']andds['foo'].resample(time='M').mean()['foo']` should be the same, but currently not

python In [16]: ds.resample(time='M').mean()['foo'] Out[16]: <xarray.DataArray 'foo' (time: 12, x: 5)> array([[-0.005705, 0.018112, 0.22818 , -0.11093 , -0.031283], [-0.007595, 0.040065, -0.099885, -0.123539, -0.013808], [ 0.112108, -0.040783, -0.023187, -0.107504, 0.082927], [-0.007728, 0.031719, 0.155191, -0.030439, 0.095658], [ 0.140944, -0.050645, 0.116619, -0.044866, -0.242026], [ 0.029198, -0.002858, 0.13024 , -0.096648, -0.170336], [-0.062954, 0.116073, 0.111285, -0.009656, -0.164599], [ 0.030806, 0.051327, -0.031282, 0.129056, -0.085851], [ 0.099617, -0.021049, 0.137962, -0.04432 , 0.050743], [ 0.117366, 0.24129 , -0.086894, 0.066012, 0.004789], [ 0.063861, -0.015472, 0.069508, 0.026725, -0.124712], [-0.058683, 0.154761, 0.028861, -0.139571, -0.037268]]) Coordinates: * time (time) datetime64[ns] 2000-01-31 2000-02-29 ... 2000-12-31 * x (x) int64 0 1 2 3 4

python In [17]: ds['foo'].resample(time='M').mean() # dimension x is gone Out[17]: <xarray.DataArray 'foo' (time: 12)> array([ 0.019675, -0.040952, 0.004712, 0.04888 , -0.015995, -0.022081, -0.00197 , 0.018811, 0.044591, 0.068512, 0.003982, -0.01038 ]) Coordinates: * time (time) datetime64[ns] 2000-01-31 2000-02-29 ... 2000-12-31

Problem description

resample should work identically for DataArray and Dataset

Expected Output

python ds.resample(time='M').mean()['foo'] == ds['foo'].resample(time='M').mean()

Code Sample, a copy-pastable example if possible

(essentially the same to #1629)

Opening netCDF file via xr.open_dataset locks a resource, preventing to write a file with the same name (as pointed out and answered as an expected behavior in #1629). ```python import xarray as xr ds = xr.Dataset({'var': ('x', [0, 1, 2])}) ds.to_netcdf('test.nc')

ds_read = xr.open_dataset('test.nc') ds.to_netcdf('test.nc') # -> PermissionError python ds_read = xr.open_dataset('test.nc').load() ds.to_netcdf('test.nc') # -> PermissionError python ds_read = xr.open_dataset('test.nc').load() ds_read.close() ds.to_netcdf('test.nc') # no error ```

Problem description

Another program cannot write the same netCDF file that xarray has opened, unless close method is not called.

-- EDIT --

close() method does not return the object, thus it cannot be put in the chain call, such as python some_function(xr.open_dataset('test.nc').close())

It is understandable when we do not want to load the entire file into the memory. However, sometimes I want to read the file that will be updated soon by another program. Also, I think that many users who are not accustomed to netCDF may expect this behavior (as np.loadtxt does) and will be surprised after getting PermissionError.

I think it would be nice to have an option such as load_all=True or even make it a default?

Expected Output

No error

Output of xr.show_versions()

Code Sample, a copy-pastable example if possible

```python import numpy as np import xarray as xr

ds = xr.DataArray(np.arange(40).reshape(8, 5), dims=['x', 'y'],
coords={'x': np.arange(8), 'y': np.arange(5)}).stack(xy=['x', 'y']) ds.isel(xy=ds['x'] < 4).unstack()

Out[1]: <xarray.DataArray (x: 8, y: 5)> array([[ 0., 1., 2., 3., 4.], [ 5., 6., 7., 8., 9.], [10., 11., 12., 13., 14.], [15., 16., 17., 18., 19.], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan]]) Coordinates: * x (x) int64 0 1 2 3 4 5 6 7 * y (y) int64 0 1 2 3 4 ```

Problem description

After unstack, there are still values that are not selected by the previous isel. Probably the upstream bug?

Expected Output

python Out[1]: <xarray.DataArray (x: 8, y: 5)> array([[ 0., 1., 2., 3., 4.], [ 5., 6., 7., 8., 9.], [10., 11., 12., 13., 14.], [15., 16., 17., 18., 19.]]) Coordinates: * x (x) int64 0 1 2 3 * y (y) int64 0 1 2 3 4

Output of xr.show_versions()

Copied from issue#2236 comments, by @st-bender

Hi, just to let you know that .std() does not accept the ddof keyword anymore (it worked in 0.10.8) Should I open a new bugreport?

Edit: It fails with:

```python ~/Work/miniconda3/envs/stats/lib/python3.6/site-packages/xarray/core/duck_array_ops.py in f(values, axis, skipna, kwargs) 234 235 try: --> 236 return func(values, axis=axis, kwargs) 237 except AttributeError: 238 if isinstance(values, dask_array_type):

TypeError: nanstd() got an unexpected keyword argument 'ddof' ```

From #2356

APIs for groupby, groupby_bins, resample, rolling are different, especially for multi-dimensional array.

```python import numpy as np import xarray as xr import pandas as pd

time = pd.date_range('2000-01-01', freq='6H', periods=365 * 4) ds = xr.Dataset({'foo': (('time', 'x'), np.random.randn(365 * 4, 5)), 'time': time, 'x': [0, 1, 2, 1, 0]})

ds.rolling(time=2).mean() # result dims : ('time', 'x') ds.resample(time='M').mean() # result dims : ('time', 'x') ds['foo'].resample(time='M').mean() # result dims : ('time', ) maybe a bug #2362 ds.groupby('time.month').mean() # result dims : ('month', ) ds.groupby_bins('time', 3).mean() # result dims : ('time_bins', ) ```

• In rolling and resample(for Dataset), reduction without argument is carried out along grouped dimension
• In rolling, reduction along other dimesnion is not possible
• In groupby and groupby_bins, reduction is applied to the grouped objects and if without argument, it reduces alongall the dimensions of each grouped object.

I think rollings API is most clean, but I am not sure it is worth to change these APIs.

The possible options would be 1. Change APIs of groupby and groupby_bins so that they share similar API with rolling. 2. Document clearly how to perform resample or groupby with multidimensional arrays.

Recently, dask renamed dask.ghost to dask.overlap. We use them around rolling. For the patch, see #2349 .

BTW, there is another faling test in python=2.7 dev, claiming that

iris and pynio which gives various errors of arrays not being equal for test_to_and_from_cdms2_sgrid and test_to_and_from_cdms2_ugrid

Is anyone working on this? If not, I think we can temporally skip these tests for python 2.7.

From SO

In the following script, the second argument is silently neglected, python da = xr.DataArray(np.random.randn(4, 3), coords={'x': [5, 7, 9, 11]}, dims=('x', 'y')) xr.apply_ufunc(np.gradient, da, da.coords['x'].values, kwargs={'axis': -1}, input_core_dims=[['x']], output_core_dims=[['x']], output_dtypes=[da.dtype]) This is because we need to the same number of input_core_dims to the number of arguments,

https://github.com/pydata/xarray/blob/56381ef444c5e699443e8b4e08611060ad5c9507/xarray/core/computation.py#L535-L538

The correct scipt might be input_core_dims=[['x']] -> input_core_dims=[['x'], []].

I think we can raise a more friendly error if the size of input_core_dims are wrong.

EDIT: Or we can automatically insert an empty tuple or None for non-xarray object? input_core_dims for non-xarray object sounds a little strange.

Some tests related to dask/distributed are failing in travis. They are raising a TypeError: can't pickle thread.lock objects.

Could anyone help to look inside?

See the travis's log for the current master: https://travis-ci.org/pydata/xarray/builds/392530577

There is an error on rtd. http://xarray.pydata.org/en/stable/plotting.html#id4

Just as a reminder of the remaining extension of #2104 .

We might add interp_like that behaves like reindex_like but using interp().

Code Sample, a copy-pastable example if possible

from #2180 , comment

python a = Dataset({ 'x': [10, 20], 'd1': ('x', [100, 200]), 'd2': ('x', [300, 400]) }) b = Dataset({ 'x': [15], 'd1': ('x', [500]), }) a.update(b) python Dataset({ 'x': [10, 20, 15], 'd1': ('x', [nan, nan, 500]), 'd2': ('x', [300, 400, nan]) })

In the above, with anything but an outer join you're destroying d2 - which doesn't even exist in the rhs dataset! A sane, desirable outcome should be

Problem description

Alignment should work

Expected Output

python Dataset({ 'x': [10, 20, 15], 'd1': ('x', [100, 200, 500]), 'd2': ('x', [300, 400, nan]) })

Code Sample, a copy-pastable example if possible

I think we can raise an Error if the newly assigned coordinate to a DataArray has an invalid shape.

python In [1]: import xarray as xr ...: ...: da = xr.DataArray([0, 1, 2], dims='x') ...: da['x'] = [0, 1, 2, 3] # no error ...: da ...: Out[1]: <xarray.DataArray (x: 3)> array([0, 1, 2]) Coordinates: * x (x) int64 0 1 2 3

Problem description

It is more user-friendly if we make some sanity checks when a new coordinate is assigned to a xr.DataArray.

Dataset raises an appropriate error, python In [2]: ds = xr.Dataset({'da': ('x', [0, 1, 2])}) ...: ds['x'] = [0, 1, 2, 3] # -> raises ValueError

Expected Output

ValueError

Code Sample, a copy-pastable example if possible

keep_attrs=True works nice for xr.DataArray, but it is neglected for xr.Variable

```python In [2]: import numpy as np In [3]: import xarray as xr In [4]: da = xr.DataArray([0, 1, 2], dims='x', attrs={'foo': 'var'}) In [5]: func = lambda x: x*2 In [6]: xr.apply_ufunc(func, da, keep_attrs=True, input_core_dims=[['x']], outpu ...: t_core_dims=[['z']]) Out[6]: <xarray.DataArray (z: 3)> # attrs are tracked for xr.DataArray array([0, 2, 4]) Dimensions without coordinates: z Attributes: foo: var

In [7]: xr.apply_ufunc(func, da.variable, keep_attrs=True, input_core_dims=[['x' ...: ]], output_core_dims=[['z']]) Out[7]: <xarray.Variable (z: 3)> # attrs are dropped array([0, 2, 4]) ```

Problem description

keep_attrs=True should work also with xr.Variable

Expected Output

python <xarray.Variable (z: 3)> # attrs are dropped array([0, 2, 4]) Attributes: foo: var

Code Sample, a copy-pastable example if possible

I noticed a bug introduced by #2087 (my PR)

```python import xarray as xr

ds = xr.Dataset({'var': ('x', [1, 2, 3])}, coords={'x': [0, 1, 2], 'z1': ('x', [1, 2, 3]), 'z2': ('x', [1, 2, 3])})

ds['var'] = ds['var'] * 2 ``` It claims a ValueError.

Problem description

https://github.com/pydata/xarray/blob/0cc64a08c672e6361d05acea3fea9f34308b62ed/xarray/core/merge.py#L564

Here should be python other[k] = obj.drop(coord_names) not python other[k] = obj.drop(*coord_names)

Code Sample, a copy-pastable example if possible

apply_ufunc can generate an invalid object if the size of the array is changed by ufunc,

python In [1]: import numpy as np ...: import xarray as xr ...: ...: da = xr.DataArray([1, 2, 3], dims=['x'], coords={'x': [0, 0.5, 1]}) ...: ...: def tile(x): ...: return np.tile(da, [2]) ...: ...: tiled = xr.apply_ufunc(tile, da, input_core_dims=[['x']], ...: output_core_dims=[['x']]) ...: tiled ...: Out[1]: <xarray.DataArray (x: 6)> array([1, 2, 3, 1, 2, 3]) Coordinates: * x (x) float64 0.0 0.5 1.0

In the above example, tiled.shape = (6, ) but tiled['x'].shape = (3,). I think we need a sanity check and drop coordinate if necessary.

Problem description

Any of our function should not generate invalid xarray objects.

Expected Output

python Out[1]: <xarray.DataArray (x: 6)> array([1, 2, 3, 1, 2, 3]) Dimensions without coordinates: x

or raise an Error.

Output of xr.show_versions()

Code Sample, a copy-pastable example if possible

```python

da = xr.DataArray([0, 1, 2], dims='x') 0 in da ( warning omitted ) False 0 in da.values True ```

Problem description

__contains__ should work as np.ndarray does.

Expected Output

```python

0 in da True ```

I found the following page fails http://xarray.pydata.org/en/stable/examples/weather-data.html

Code Sample, a copy-pastable example if possible

I sometimes want to make more flexible dot product of two data arrays, where we sum up along a part of common dimensions.

```python

Your code here

da_vals = np.arange(6 * 5 * 4).reshape((6, 5, 4)) da = DataArray(da_vals, dims=['x', 'y', 'z']) dm_vals = np.arange(6 * 4).reshape((6, 4)) dm = DataArray(dm_vals, dims=['x', 'z'])

I want something like this

da.dot(dm, 'z') # -> dimensions of the output array: ['x', 'y'] ```

It's an intermediate path of np.matmul and np.tensordot.

Is this feature sufficiently universal?

EDIT: I just noticed dask does not have einsum yet (dask/dask#732). We maybe need to wait or decide to support only numpy arrays.

Problem description

Tests are failing due to the release of zarr 2.2.0

Travis's log https://travis-ci.org/pydata/xarray/jobs/351566529

I noticed the failing in rtd http://xarray.pydata.org/en/stable/computation.html#missing-values is because it still uses numpy=1.11 which does not support signature argument for np.vectorize.

This can be easily fixed (just bumping up numpy's version on rtd), but as our minimum requirement is numpy==1.11, we may need to take care of this in xr.apply_ufunc.

Code Sample, a copy-pastable example if possible

```python In [1]: import numpy as np ...: import xarray as xr ...: ...: da = xr.DataArray(np.random.randn(1000, 100), dims=['x', 'y'], ...: coords={'x': np.arange(1000)}) ...:

In [2]: %%timeit ...: da.rolling(x=10).reduce(np.sum) ...: 2.04 s ± 8.25 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) ```

Problem description

In DataArray.rolling, we index by .isel method for every window, constructing huge number of xr.DataArray instances. This is very inefficient.

Of course, we can use bottleneck methods if available, but this provides only a limited functions. (This also limits possible extensions of rolling, such as ND-rolling (#819), window type (#1142), strides (#819).)

I am wondering if we could skip any sanity checks in our DataArray.isel -> Variable.isel path in indexing. Or can we directly construct a single large DataArray instead of a lot of small DataArrays?

From gitter cc. @davidh-ssec

python In [1]: import numpy as np ...: import xarray as xr ...: ...: da = xr.DataArray(np.arange(100), dims='x').chunk({'x': 10}) ...: da.isnull() ...: Out[1]: <xarray.DataArray (x: 100)> array([False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], dtype=bool) Dimensions without coordinates: x

Problem description

DataArray.isnull() automatically computes dask arrays.

Expected Output

python Out[2]: <xarray.DataArray (x: 100)> dask.array<shape=(100,), dtype=bool, chunksize=(10,)> Dimensions without coordinates: x

Cause

https://github.com/pydata/xarray/blob/697cc74b9af5fbfedadd54fd07019ce7684553ec/xarray/core/ops.py#L322-L324

Here, getattr(pd, name) should be getattr(duck_array_ops, name).

```python In [1]: import xarray as xr ...: import numpy as np ...: ...: a = np.arange(10) ...: a[np.array(0)] ...: Out[1]: 0

In [2]: da = xr.DataArray(a, dims='x') ...: da[np.array(0)] ...:

TypeError Traceback (most recent call last) <ipython-input-2-d30fdfc612ec> in <module>() 1 da = xr.DataArray(a, dims='x') ----> 2 da[np.array(0)]

/home/keisukefujii/Dropbox/projects/xarray.git/xarray/core/dataarray.pyc in getitem(self, key) 478 else: 479 # xarray-style array indexing --> 480 return self.isel(**self._item_key_to_dict(key)) 481 482 def setitem(self, key, value):

/home/keisukefujii/Dropbox/projects/xarray.git/xarray/core/dataarray.pyc in isel(self, drop, indexers) 759 DataArray.sel 760 """ --> 761 ds = self._to_temp_dataset().isel(drop=drop, indexers) 762 return self._from_temp_dataset(ds) 763

/home/keisukefujii/Dropbox/projects/xarray.git/xarray/core/dataset.py in isel(self, drop, indexers) 1390 for name, var in iteritems(self._variables): 1391 var_indexers = {k: v for k, v in indexers_list if k in var.dims} -> 1392 new_var = var.isel(var_indexers) 1393 if not (drop and name in var_indexers): 1394 variables[name] = new_var

/home/keisukefujii/Dropbox/projects/xarray.git/xarray/core/variable.pyc in isel(self, **indexers) 851 if dim in indexers: 852 key[i] = indexers[dim] --> 853 return self[tuple(key)] 854 855 def squeeze(self, dim=None):

/home/keisukefujii/Dropbox/projects/xarray.git/xarray/core/variable.pyc in getitem(self, key) 619 array x.values directly. 620 """ --> 621 dims, indexer, new_order = self._broadcast_indexes(key) 622 data = as_indexable(self._data)[indexer] 623 if new_order:

/home/keisukefujii/Dropbox/projects/xarray.git/xarray/core/variable.pyc in _broadcast_indexes(self, key) 477 # key can be mapped as an OuterIndexer. 478 if all(not isinstance(k, Variable) for k in key): --> 479 return self._broadcast_indexes_outer(key) 480 481 # If all key is 1-dimensional and there are no duplicate labels,

/home/keisukefujii/Dropbox/projects/xarray.git/xarray/core/variable.pyc in _broadcast_indexes_outer(self, key) 542 new_key.append(k) 543 --> 544 return dims, OuterIndexer(tuple(new_key)), None 545 546 def _nonzero(self):

/home/keisukefujii/Dropbox/projects/xarray.git/xarray/core/indexing.py in init(self, key) 368 raise TypeError('invalid indexer array for {}, must have ' 369 'exactly 1 dimension: ' --> 370 .format(type(self).name, k)) 371 k = np.asarray(k, dtype=np.int64) 372 else:

TypeError: invalid indexer array for OuterIndexer, must have exactly 1 dimension: ```

Indexing by a 0d-array should be identical to the indexing by a scalar.

This issue arises in #1837 comment, where we need to make a summation of object-dtype array, such as python xr.DataArray(np.array([True, True, False, np.nan], dtype=object), dims='x').sum('x', skipna=True) Currently, it raises NotImplementedError.

pandas support this by having their own nan-aggregation methods.

In working with #1883, I noticed that our (and numpy's) std and var work differently from pandas. Pandas assumes ddof=1 as default, while we assume ddof=0. Do we want to match these behaviors to pandas?

```python In [1]: import numpy as np ...: import xarray as xr ...: da = xr.DataArray([0, 1, 2], dims='x', name='da')

In [2]: da.std() Out[2]: <xarray.DataArray 'da' ()> array(0.816496580927726)

In [3]: da.to_dataframe().std() Out[3]: da 1.0 dtype: float64

In [4]: da.std(ddof=1) Out[4]: <xarray.DataArray 'da' ()> array(1.0) ```

I saw the coverage badge on our README with only 75% (true value 95%), which may be caused by github page caching.

Similar issues seem frequently reported, e.g. https://github.com/codecov/support/issues/218 Maybe we can improve this, though it would be difficult to test...

Code Sample, a copy-pastable example if possible

```python

Your code here

import xarray as xr ds = xr.Dataset({'a': ('x', [0, 1, 2])}) ds. -> press 'Tab' ```

Problem description

IPython autocomplete raises a deprecation warning, introducing in #1643. ipython /home/keisukefujii/anaconda3/envs/tensorflow/lib/python3.5/site-packages/jedi/e. getattr(obj, name) /home/keisukefujii/anaconda3/envs/tensorflow/lib/python3.5/site-packages/jedi/e. obj = getattr(obj, name) In [3]: ds.

Expected Output

None

Output of xr.show_versions()

found some outdated parts in docs

• In FAQ related to pandas N-dimensional panel It is deprecated and the description may be better to change.

• In FAQ related to Zendo The link of the badge is outdated.

Although we updated pandas minimum version to 0.18 in #1530 , we still check its version like pd.__version__ >= '0.17' in some tests.

(I forgot to remove these statements in #1530 .)

We need pandas >= 0.18 because dataframe.rolling is supported after 0.18. But requirements in our setup.py says we need pandas >= 0.15.

Additionally, I noticed that in travis's CONDA_ENV=py27-min setup, our unit tests run with pandas == 0.20, though it might be intended to run with pandas == 0.15.

By conda remove scipy, pandas.0.15 is removed. (Here is the travis log) ``` if [[ "$CONDA_ENV" == "py27-min" ]]; then conda remove scipy; fi Fetching package metadata ......... Solving package specifications: .

Package plan for package removal in environment /home/travis/miniconda/envs/test_env:

The following packages will be REMOVED:

pandas: 0.15.0-np19py27_0 defaults
scipy:  0.17.1-np19py27_1 defaults

then in `python setup.py install`, pandas==0.20.3 is installed. Searching for pandas>=0.15.0 Reading https://pypi.python.org/simple/pandas/ Downloading https://pypi.python.org/packages/ee/aa/90c06f249cf4408fa75135ad0df7d64c09cf74c9870733862491ed5f3a50/pandas-0.20.3.tar.gz#md5=4df858f28b4bf4fa07d9fbb7f2568173 Best match: pandas 0.20.3 ```

Boolean indexing for np.ndarray 'by' a boolean xr.DataArray behaves strange, ```python In [1]: import numpy as np ...: import xarray as xr ...: ...: ind = xr.DataArray([True, True, False], dims=['x']) ...: ind Out[1]: <xarray.DataArray (x: 3)> array([ True, True, False], dtype=bool) Dimensions without coordinates: x

In [2]: np.arange(3)[ind.values] Out[2]: array([0, 1])

In [3]: np.arange(3)[ind] Out[3]: array([], shape=(0, 3), dtype=int64) ``` (This is numpy==1.13)

numpy==1.11 behaves differently, ```python

np.arange(3)[ind.values] array([0, 1])

np.arange(3)[ind] main:1: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future Traceback (most recent call last): File "<stdin>", line 1, in <module> IndexError: too many indices for array ```

There are some typos in reshaping.rst, that is newly added in #1347.

Based on the post http://stackoverflow.com/questions/34987972/expand-dimensions-xray and issue #1323.

I think it would be great if xarray has a method to expand dimension, such as expand_dims in numpy. (xr.DataArray.squeeze is already implemented.)

The positional indexing fails if the coordinate is large np.float32 array. The minimum working example is

da_float64 = xr.DataArray(np.random.randn(1000000), coords={'Time': np.linspace(0.0,1.0,1000000).astype(np.float64)}) da_float64.loc[0.2:0.3] # This works fine

da_float32 = xr.DataArray(np.random.randn(1000000), coords={'Time': np.linspace(0.0,1.0,1000000).astype(np.float32)}) da_float32.loc[0.2:0.3] # This causes a kernel freeze.

With a smaller sized DataArray, both works.

da_float32.to_pandas()

also freezes the kernel.

Are these a pandas's issue?

I used Python 3.5 and xarray 0.8.2.