Ported from #1724, comment by @shoyer

In the long term, it would be nice to get ride of these uses of _data, maybe by switching entirely from our lazy array classes to Dask.

The subtleties of checking _data vs data are undesirable, e.g., consider the bug on these lines: https://github.com/pydata/xarray/blob/1a012080e0910f3295d0fc26806ae18885f56751/xarray/core/formatting.py#L212-L213

• [x] Closes #3868
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Now we use a tuple of indexes for DataArray.pad and Dataset.pad.

• [x] Closes #3562
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Now 0d-DataArray can be used for indexing.

MCVE Code Sample

I want to minimize the number of calls .item() within my data analysis. It often happens

1. when putting a 0d-DataArray into a slice python da = xr.DataArray([0.5, 4.5, 2.5], dims=['x'], coords={'x': [0, 1, 2]}) da[: da.argmax()] -> TypeError: 'DataArray' object cannot be interpreted as an integer

2. when using a 0d-DataArray for selecting python da = xr.DataArray([0.5, 4.5, 2.5], dims=['x'], coords={'x': [0, 0, 2]}) da.sel(x=da['x'][0]) -> IndexError: arrays used as indices must be of integer (or boolean) type

Both cases, I need to call '.item()'. It is not a big issue, but I think it would be nice if xarray becomes more self-contained.

In #4219 we added ndrolling. However, nanreduce, such as ds.rolling(x=3, y=2).mean() calls np.nanmean which copies the strided-array into a full-array. This is memory-inefficient.

We can implement inhouse-nanreduce methods for the strided array. For example, our .nansum currently does make a strided array -> copy the array -> replace nan by 0 -> sum but we can do instead replace nan by 0 -> make a strided array -> sum This is much more memory efficient.

I think we can continue the discussion we have in #1426 about MultiIndex here.

In comment , @shoyer recommended to remove MultiIndex from public API.

I agree with this, as long as my codes work with this improvement.

I think if we could have a list of possible MultiIndex use cases here, it would be easier to deeply discuss and arrive at a consensus of the future API.

Current limitations of MultiIndex are + It drops scalar coordinate after selection #1408, #1491 + It does not support to serialize to NetCDF #1077 + Stack/unstack behaviors are inconsistent #1431

• [x] Closes #2007, #2011
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Added some boundary options for rolling.construct. Currently, the option names are inherited from np.pad, ['edge' | 'reflect' | 'symmetric' | 'wrap']. Do we want a more intuitive name, such as periodic?

I am sorry but I unintentionally pushed my working scripts to xarray.master. (I thought it is not allowed and I was not careful.)

What is the best way to reset this? I'm thinking to do in my local, and force push again, but I'm afraid that I do another wrong thing...

I apologize for my mistake.

np.nonzero to DataArray returns a wrong result,

python In [4]: da = xr.DataArray(np.arange(12).reshape(4, 3), dims=['x', 'y'], ...: coords={'x': [0, 1, 2, 3], 'y': ['a', 'b', 'c']}) ...: np.nonzero(da) ...: Out[4]: <xarray.DataArray (x: 2, y: 11)> array([[0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3], [1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2]]) Coordinates: * x (x) int64 0 1 2 3 * y (y) <U1 'a' 'b' 'c'

Problem description

Apparently, the dimensions and the coordinates conflict each other. I think we can have our own nonzero method, which may return a Dataset consisting of indexes and appropriate coordinates.

Output of xr.show_versions()

groupby works inconsistently depending on the dimension order of a DataArray. Furthermore, in some cases, this causes a corrupted object.

python In [4]: data = xr.DataArray( ...: np.random.randn(4, 2), ...: dims=['x', 'z'], ...: coords={'x': ['a', 'b', 'a', 'c'], 'y': ('x', [0, 1, 0, 2])} ...: ) ...: ...: data.groupby('x').mean() Out[4]: <xarray.DataArray (x: 3, z: 2)> array([[ 0.95447186, -1.14467028], [ 0.76294958, 0.3751244 ], [-0.41030223, -1.35344548]]) Coordinates: * x (x) object 'a' 'b' 'c' Dimensions without coordinates: z groupby works fine (although this drops nondimensional coordinate y, related to #3745).

However, groupby does not give a correct result if we work on the second dimension, python In [5]: data.T.groupby('x').mean() # <--- change the dimension order, and do the same thing Out[5]: <xarray.DataArray (z: 2, x: 3)> array([[ 0.95447186, 0.76294958, -0.41030223], [-1.14467028, 0.3751244 , -1.35344548]]) Coordinates: * x (x) object 'a' 'b' 'c' y (x) int64 0 1 0 2 # <-- the size must be 3!! Dimensions without coordinates: z

The bug has been discussed in #2944 and solved, but I found this is still there.

.sel method usually keeps selected coordinate value as a scalar coordinate

```python In[4] ds1 = xr.Dataset({'foo': (('x',), [1, 2, 3])}, {'x': [1, 2, 3], 'y': 'a'})

Out[4]: <xarray.Dataset> Dimensions: () Coordinates: y <U1 'a' x int64 1 Data variables: foo int64 1 ```

But in MultiIndex case, does not.

python In[5] ds2 = xr.Dataset({'foo': (('x',), [4, 5, 6])}, {'x': [1, 2, 3], 'y': 'b'}) ds = xr.concat([ds1, ds2], dim='y').stack(yx=['y', 'x']) ds.sel(x=2) Out[5] <xarray.Dataset> Dimensions: (y: 2) Coordinates: * y (y) object 'a' 'b' Data variables: foo (y) int64 2 5 x is gone. Is it a desired behavior?

We have a bunch of terms to describe the xarray structure, such as dimension, coordinate, dimension coordinate, etc.. Although it has been discussed in #1295 and we tried to use the consistent terminology in our docs, it looks still not easy for users to understand our functionalities.

In #2399, @horta wrote a list of definitions (https://drive.google.com/file/d/1uJ_U6nedkNe916SMViuVKlkGwPX-mGK7/view?usp=sharing). I think it would be nice to have something like this in our docs.

Any thought?

Code Sample, a copy-pastable example if possible

```python

import numpy as np import xarray as xr

da = xr.DataArray(np.arange(12).reshape(3, 4), dims=['x', 'y'], ... coords={'x': [0, 1, 1], 'x2': ('x', ['a', 'b', 'c'])}) grouped = da.groupby('x').mean('x') grouped <xarray.DataArray (x: 2, y: 4)> array([[0., 1., 2., 3.], [6., 7., 8., 9.]]) Coordinates: * x (x) int64 0 1 x2 (x) <U1 'a' 'b' 'c' # <-- this has length 3! Dimensions without coordinates: y ```

Problem description

groupby operation ignores the non-dimensional coordinate.

Expected Output

python <xarray.DataArray (x: 2, y: 4)> array([[0., 1., 2., 3.], [6., 7., 8., 9.]]) Coordinates: * x (x) int64 0 1 Dimensions without coordinates: y Compute mean of the coordinate. If not possible, drop it.

From a discussion in #1473 comment

It would be convenient if we have multi-dimensional reindex method, where we consider dimensions and coordinates of indexers. The proposed outline by @shoyer is

• Given reindex arguments of the form dim=array where array is a 1D unlabeled array/list, convert them into DataArray(array, [(dim, array)]).
• Do multi-dimensional indexing with broadcasting like sel, but fill in NaN for missing values (we could allow for customizing this with a fill_value argument).
• Join coordinates like for sel, but coordinates from the indexers take precedence over coordinates from the object being indexed.

Currently I'm using xarray to handle multiple images (typically, a sequence of images), and I feel it would be convenient if xarray supports image related functions.

There may be many possibilities, but particular methods I want to have in xarray are 1. xr.open_image(File)
Currently, I open image by PILLOW, convert to np.ndarray, extract its attributes, then construct xr.DataArray from them. If I can do it by 1 line, it would be very great.

1. xr.DataArray.expand_dims(dim) I want to add additional channel dimension to grey scale images (size [W x H] -> [W x H x 1]), in order to pass them into convolutional neural network, which usually accepts 4-dimensional tensor [Batch x W x H x channel].

Image (possibly also video?) is naturally high-dimensional and I guess it would fit xarray's concept. Is this sufficiently broad interest?

• [x] Closes #4328
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8

There was a bug in rolling.__repr__ but it was not tested. Fixed and tests are added.

• [x] Closes #4196
• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

I noticed that the implementation of nd-rolling is straightforward. The core part is implemented but I am wondering what the best API is, with keeping it backward-compatible.

Obviously, it is basically should look like python da.rolling(x=3, y=3).mean()

A problem is other parameters, centers and min_periods. In principle, they can depend on dimension. For example, we can have center=True only for x but not for y.

So, maybe we allow dictionary for them? python da.rolling(x=3, y=3, center={'x': True, 'y': False}, min_periods={'x': 1, 'y': None}).mean()

The same thing happens for .construct method. python da.rolling(x=3, y=3).construct(x='x_window', y='y_window', stride={'x': 2, 'y': 1}) I'm afraid if this dictionary argument was a bit too redundant.

Does anyone have another idea?

I noticed that some users experiencing troubles to restore xarray objects that is created xarray < 0.8.

Is there any possibility to add a function to support unpickling old object, such as xr.unpickle_legacy(file, version)? We previously recommended using pickle to store xarray objects (at least for the short term use, maybe).

xref (private repo) gafusion/OMFIT-source#2652

• [x] Closes #1388
• [x] Tests added / passed
• [x] Passes git diff master | flake8 --diff
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

With this PR, ValueError raises if argmin() is called by a multi-dimensional array.

argmin_indexes() method is also added for xr.DataArray. Current API design for argmin_indexes() returns the argmin-indexes as an OrderedDict of DataArrays.

Example: ```python

In [1]: import xarray as xr ...: da = xr.DataArray([[1, 2], [-1, 40], [5, 6]], ...: [('x', ['c', 'b', 'a']), ('y', [1, 0])]) ...: ...: da.argmin_indexes() ...: Out[1]: OrderedDict([('x', <xarray.DataArray 'x' ()> array(1)), ('y', <xarray.DataArray 'y' ()> array(0))])

In [2]: da.argmin_indexes(dims='y') Out[2]: OrderedDict([('y', <xarray.DataArray 'y' (x: 3)> array([0, 0, 0]) Coordinates: * x (x) <U1 'c' 'b' 'a')])

```

(Because the returned object is an OrderedDict, it is not beautifully printed. The returned type can be a xr.Dataset if we want.)

Although in #1388 argmin_indexes() was originally suggested so that we can pass the result into isel_point,

python da.isel_points(**da.argmin_indexes()) current implementation of isel_points does NOT work for this case.

This is mainly because 1. isel_points currently does not work for 0-dimensional or multi-dimensional input. 2. Even for 1-dimensional input (the second one in the above examples), we should also pass x as an indexer rather than the coordinate of indexer.

For 1, I have prepared modification of isel_points to accept multi-dimensional arrays, but I guess it should be in another PR after the API decision. (It is related in #475, and #974.)

For 2, we should either + change API of argmin_indexes to return not only the indicated dimension but also all the dimensions, like

```python In [2]: da.argmin_indexes(dims='y') Out[2]: OrderedDict([('y', array([0, 0, 0]), 'x', array(['c' 'b' 'a']))

`` or + change API ofisel_pointso that it takes care of theindexer's coordinate ifxr.DataArrayis passed for asindexers`.

I originally worked with the second option for the modification of isel_points, the second option breaks the backward-comaptibility and is somehow magical.

Another alternertive is to + change API of argmin_indexes to return xr.Dataset rather than an OrderedDict, and also change API of isel_points to accept xr.Dataset.
It keeps backward-compatibility.

Any comments are welcome.

• [x] Closes #2223
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Now n-dimensional interp works sequentially if possible. It may speed up some cases.

Currently, we disallow to save complex values to a netCDF file. Maybe netCDF itself does not support complex values, but there may be some workarounds. It would be very handy for me.

The most naive workaround may be to split each complex value into a real and imaginary part, add some flags, and restore it when loading them from the file. Maybe we may add a special suffix to the variable name? ```python

ds = xr.Dataset({'a': ('x': [1+2j, 2+3j])}, coords={'x': [0, 1]}) ds.to_netcdf('tmp.nc', encode_complex=True) xr.load_netcdf('tmp.nc') <xarray.Dataset> Dimensions: (x: 2) Coordinates: * x (x) int64 0 1 Data variables: a__real__ (x) int64 1 2 a__imag__ (x) int64 2 3 xr.load_netcdf('tmp.nc', decode_complex=True) <xarray.Dataset> Dimensions: (x: 2) Coordinates: * x (x) int64 0 1 Data variables: a (x) complex128 (1+2j) (2+3j) ```

I think there may be a better way. Any thoughts are welcome :)

p.s. I just found that engine=h5netcdf can save complex values, but the file becomes an invalid netcdf file. I'm not sure if it worth the trouble just to make a valid netCDF file.

• [x] Closes #4024
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Now it looks like

I'm pretty sure that this workaround is not the best (maybe the second worst), as it only supports the dark mode of vscode but not other environments.

I couldn't find a good way to make a workaround for the general dark-mode. Any advice is welcome.

If using xarray inside VScode with darkmode, the new html repr has a small contrast of the text color and background.

Maybe the text color comes from the default setting, but the background color is not. In light mode, it looks nice.

Versions

• [x] Closes #3951
• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

It looks dataframe.set_index(index).index == index is not always true.

Added a workaround for this...

Related to #138

Why is keep_attrs option in reduce method set to FALSE by default? I feel it is more natural to keep all the attrs after some computation that returns xr.DaraArray Such as data*100. (By it is not possible to set this option TRUE when using an operator.) Is it an option to move this option to init method, in case of TRUE all the attrs are tracked after computations of the object and also the object generated from this object?

• [x] Closes #3669, #3674
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

It is a bit surprising that no members have used xarray with CategoricalIndex... If there is anything missing additionally, please feel free to point it out.

• [x] Closes #3518
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Added sparse option to reindex and unstack. I just added a minimal set of codes necessary to unstack and reindex.

There is still a lot of space to complete the sparse support as discussed in #3245.

• [x] Closes #3518
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Added an option fill_value for unstack. I am trying to add sparse option too, but it may take longer. Probably better to do in a separate PR?

• [x] Closes #3512
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

There was a bug in set_index, where an old dimension was not updated if it becomes a level of MultiIndex.

MCVE Code Sample

```python da = xr.DataArray([0, 1], dims=['x'], coords={'x': [0, 1], 'y': 'a'}) db = xr.DataArray([2, 3], dims=['x'], coords={'x': [0, 1], 'y': 'b'}) data = xr.concat([da, db], dim='x').set_index(xy=['x', 'y']) data.sel(y='a')

<xarray.DataArray (x: 4)> array([0, 1, 2, 3]) Coordinates: * x (x) int64 0 1 ```

Expected Output

```python

<xarray.DataArray (x: 2)> array([0, 1]) Coordinates: * x (x) int64 0 1 ```

Problem Description

Should select the array

Output of xr.show_versions()

Sorry for being quiet for a long time. I hope I could send a fix for this in a few days...

Tests are failing, which is caused by new release of dask==0.18.2. xref: dask/dask#3822

https://github.com/pydata/xarray/blob/5e801894886b2060efa8b28798780a91561a29fd/xarray/core/dataarray.py#L482-L489

In #1746, we added a validation in xr.DataArray.__setitem__ whether the coordinates consistency of array, key, and values are checked. In the current implementation, we call xr.DataArray.__getitem__ to use the existing coordinate validation logic, but it does unnecessary indexing and it may decrease the __setitem__ performance if the arrray is multidimensional.

We may need to optimize the logic here.

Is it reasonable to constantly monitor the performance of basic operations, such as Dataset construction, alignment, indexing, and assignment? (or are these operations too light to make a performance monitor?)

cc @jhamman @shoyer

From #2356, I noticed resample and groupby works nice for Dataset but not for DataArray

Code Sample, a copy-pastable example if possible

```python In [14]: import numpy as np ...: import xarray as xr ...: import pandas as pd ...: ...: time = pd.date_range('2000-01-01', freq='6H', periods=365 * 4) ...: ds = xr.Dataset({'foo': (('time', 'x'), np.random.randn(365 * 4, 5)), 'time': time, ...: 'x': np.arange(5)})

In [15]: ds Out[15]: <xarray.Dataset> Dimensions: (time: 1460, x: 5) Coordinates: * time (time) datetime64[ns] 2000-01-01 ... 2000-12-30T18:00:00 * x (x) int64 0 1 2 3 4 Data variables: foo (time, x) float64 -0.6916 -1.247 0.5376 ... -0.2197 -0.8479 -0.6719 ``ds.resample(time='M').mean()['foo']andds['foo'].resample(time='M').mean()['foo']` should be the same, but currently not

python In [16]: ds.resample(time='M').mean()['foo'] Out[16]: <xarray.DataArray 'foo' (time: 12, x: 5)> array([[-0.005705, 0.018112, 0.22818 , -0.11093 , -0.031283], [-0.007595, 0.040065, -0.099885, -0.123539, -0.013808], [ 0.112108, -0.040783, -0.023187, -0.107504, 0.082927], [-0.007728, 0.031719, 0.155191, -0.030439, 0.095658], [ 0.140944, -0.050645, 0.116619, -0.044866, -0.242026], [ 0.029198, -0.002858, 0.13024 , -0.096648, -0.170336], [-0.062954, 0.116073, 0.111285, -0.009656, -0.164599], [ 0.030806, 0.051327, -0.031282, 0.129056, -0.085851], [ 0.099617, -0.021049, 0.137962, -0.04432 , 0.050743], [ 0.117366, 0.24129 , -0.086894, 0.066012, 0.004789], [ 0.063861, -0.015472, 0.069508, 0.026725, -0.124712], [-0.058683, 0.154761, 0.028861, -0.139571, -0.037268]]) Coordinates: * time (time) datetime64[ns] 2000-01-31 2000-02-29 ... 2000-12-31 * x (x) int64 0 1 2 3 4

python In [17]: ds['foo'].resample(time='M').mean() # dimension x is gone Out[17]: <xarray.DataArray 'foo' (time: 12)> array([ 0.019675, -0.040952, 0.004712, 0.04888 , -0.015995, -0.022081, -0.00197 , 0.018811, 0.044591, 0.068512, 0.003982, -0.01038 ]) Coordinates: * time (time) datetime64[ns] 2000-01-31 2000-02-29 ... 2000-12-31

Problem description

resample should work identically for DataArray and Dataset

Expected Output

python ds.resample(time='M').mean()['foo'] == ds['foo'].resample(time='M').mean()

• [x] Closes #1775 (remove if there is no corresponding issue, which should only be the case for minor changes)
• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Passes git diff upstream/master **/*py | flake8 --diff (remove if you did not edit any Python files)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

I am trying to fix #1775, but tests are still failing. Any help would be appreciated.

• [x] Closes #2940
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Fix for #2940 It looks that there was a bug in the previous logic, but I am not sure why it was working...

Code Sample, a copy-pastable example if possible

(essentially the same to #1629)

Opening netCDF file via xr.open_dataset locks a resource, preventing to write a file with the same name (as pointed out and answered as an expected behavior in #1629). ```python import xarray as xr ds = xr.Dataset({'var': ('x', [0, 1, 2])}) ds.to_netcdf('test.nc')

ds_read = xr.open_dataset('test.nc') ds.to_netcdf('test.nc') # -> PermissionError python ds_read = xr.open_dataset('test.nc').load() ds.to_netcdf('test.nc') # -> PermissionError python ds_read = xr.open_dataset('test.nc').load() ds_read.close() ds.to_netcdf('test.nc') # no error ```

Problem description

Another program cannot write the same netCDF file that xarray has opened, unless close method is not called.

-- EDIT --

close() method does not return the object, thus it cannot be put in the chain call, such as python some_function(xr.open_dataset('test.nc').close())

It is understandable when we do not want to load the entire file into the memory. However, sometimes I want to read the file that will be updated soon by another program. Also, I think that many users who are not accustomed to netCDF may expect this behavior (as np.loadtxt does) and will be surprised after getting PermissionError.

I think it would be nice to have an option such as load_all=True or even make it a default?

Expected Output

No error

Output of xr.show_versions()

• [x] Closes #2667
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Started to fixing #2667

• [x] Closes #1288
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

I would like to add integrate, which is essentially an xarray-version of np.trapz. I know there was variety of discussions in #1288, but I think it would be nice to limit us within that numpy provides by np.trapz, i.e., 1. only for trapz not rectangle or simps 2. do not care np.nan 3. do not support bounds Most of them (except for 1) can be solved by combining several existing methods.

• [x] Closes #1408
• [x] Tests added / passed
• [x] Passes git diff upstream/master | flake8 --diff
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

[Edit for more clarity] I restarted a new branch to fix #1408 (I closed the older one #1412).

Because the changes I made is relatively large, here I summarize this PR.

Sumamry

In this PR, I newly added two kinds of levels in MultiIndex, index-level and scalar-level. index-level is an ordinary level in MultiIndex (as in current implementation), while scalar-level indicates dropped level (which is newly added in this PR).

Changes in behaviors.

1. Indexing a scalar at a particular level changes that level to scalar-level instead of dropping that level (changed from #767).
2. Indexing a scalar from a MultiIndex, the selected value now becomes a MultiIndex-scalar rather than a scalar of tuple.
3. Enabled indexing along a index-level if the MultiIndex has only a single index-level.

Examples of the output are shown below. Any suggestions for these behaviors are welcome.

```python In [1]: import numpy as np ...: import xarray as xr ...: ...: ds1 = xr.Dataset({'foo': (('x',), [1, 2, 3])}, {'x': [1, 2, 3], 'y': 'a'}) ...: ds2 = xr.Dataset({'foo': (('x',), [4, 5, 6])}, {'x': [1, 2, 3], 'y': 'b'}) ...: # example data ...: ds = xr.concat([ds1, ds2], dim='y').stack(yx=['y', 'x']) ...: ds Out[1]: <xarray.Dataset> Dimensions: (yx: 6) Coordinates: * yx (yx) MultiIndex - y (yx) object 'a' 'a' 'a' 'b' 'b' 'b' # <--- this is index-level - x (yx) int64 1 2 3 1 2 3 # <--- this is also index-level Data variables: foo (yx) int64 1 2 3 4 5 6

In [2]: # 1. indexing a scalar converts index-level x to scalar-level. ...: ds.sel(x=1) Out[2]: <xarray.Dataset> Dimensions: (yx: 2) Coordinates: * yx (yx) MultiIndex - y (yx) object 'a' 'b' # <--- this is index-level - x int64 1 # <--- this is scalar-level Data variables: foo (yx) int64 1 4

In [3]: # 2. indexing a single element from MultiIndex makes a MultiIndex-scalar ...: ds.isel(yx=0) Out[3]: <xarray.Dataset> Dimensions: () Coordinates: yx MultiIndex # <--- this is MultiIndex-scalar - y <U1 'a' - x int64 1 Data variables: foo int64 1

In [6]: # 3. Enables to selecting along a index-level if only one index-level exists in MultiIndex ...: ds.sel(x=1).isel(y=[0,1]) Out[6]: <xarray.Dataset> Dimensions: (yx: 2) Coordinates: * yx (yx) MultiIndex - y (yx) object 'a' 'b' - x int64 1 Data variables: foo (yx) int64 1 4

```

Changes in the public APIs

Some changes were necessary to the public APIs, though I tried to minimize them.

• level_names, get_level_values methods were moved from IndexVariable to Variable. This is because IndexVariable cannnot handle 0-d array, which I want to support in 2.

• scalar_level_names and all_level_names properties were added to Variable

• reset_levels method was added to Variable class to control scalar-level and index-level.

Implementation summary

The main changes in the implementation is the addition of our own wrapper of pd.MultiIndex, PandasMultiIndexAdapter. This does most of MultiIndex-related operations, such as indexing, concatenation, conversion between 'scalar-levelandindex-level`.

What we can do now

The main merit of this proposal is that it enables us to handle MultiIndex more consistent way to the normal Variable. Now we can

• recover the MultiIndex with dropped level. ```python In [5]: ds.sel(x=1).expand_dims('x') Out[5]: <xarray.Dataset> Dimensions: (yx: 2) Coordinates:
• yx (yx) MultiIndex
• y (yx) object 'a' 'b'

• x (yx) int64 1 1 Data variables: foo (yx) int64 1 4 ```

• construct a MultiIndex by concatenation of MultiIndex-scalar. ```python In [8]: xr.concat([ds.isel(yx=i) for i in range(len(ds['yx']))], dim='yx') Out[8]: <xarray.Dataset> Dimensions: (yx: 6) Coordinates:

• yx (yx) MultiIndex
• y (yx) object 'a' 'a' 'a' 'b' 'b' 'b'
• x (yx) int64 1 2 3 1 2 3 Data variables: foo (yx) int64 1 2 3 4 5 6 ```

What we cannot do now

With the current implementation, we can do python ds.sel(y='a').rolling(x=2) but with this PR we cannot, because x is not yet an ordinary coordinate, but a MultiIndex with a single index-level. I think it is better if we can handle such a MultiIndex with a single index-level as very similar way to an ordinary coordinate.

Similary, we can neither do ds.sel(y='a').mean(dim='x'). Also, ds.sel(y='a').to_netcdf('file') (#719)

What are to be decided

• How to repr these new levels (Current formatting is shown in Out[2] and Out[3] above.)
• Terminologies such as index-level, scalar-level, MultiIndex-scalar are clear enough?
• How much operations should we support for a single index-level MultiIndex? Do we support ds.sel(y='a').rolling(x=2) and ds.sel(y='a').mean(dim='x')?

TODOs

• [ ] Support indexing with DataAarray, ds.sel(x=ds.x[0])
• [ ] Support stack, unstack, set_index, reset_index methods with scalar-level MultiIndex.
• [ ] Add a full document
• [ ] Clean up the code related to MultiIndex
• [ ] Fix issues (#1428, #1430, #1431) related to MultiIndex

• [x] Closes #2525
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Started to implement corsen. The API is currently something like python actual = ds.coarsen(time=2, x=3, side='right', coordinate_func={'time': np.max}).max()

Currently, it is not working for a datetime coordinate, since mean does not work for this dtype. e.g. python da = xr.DataArray(np.linspace(0, 365, num=365), dims='time', coords={'time': pd.date_range('15/12/1999', periods=365)}) da['time'].mean() # -> TypeError: ufunc add cannot use operands with types dtype('<M8[ns]') and dtype('<M8[ns]')

I am not familiar with datetime things. Any advice will be appreciated.

Code Sample, a copy-pastable example if possible

```python import numpy as np import xarray as xr

ds = xr.DataArray(np.arange(40).reshape(8, 5), dims=['x', 'y'],
coords={'x': np.arange(8), 'y': np.arange(5)}).stack(xy=['x', 'y']) ds.isel(xy=ds['x'] < 4).unstack()

Out[1]: <xarray.DataArray (x: 8, y: 5)> array([[ 0., 1., 2., 3., 4.], [ 5., 6., 7., 8., 9.], [10., 11., 12., 13., 14.], [15., 16., 17., 18., 19.], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan]]) Coordinates: * x (x) int64 0 1 2 3 4 5 6 7 * y (y) int64 0 1 2 3 4 ```

Problem description

After unstack, there are still values that are not selected by the previous isel. Probably the upstream bug?

Expected Output

python Out[1]: <xarray.DataArray (x: 8, y: 5)> array([[ 0., 1., 2., 3., 4.], [ 5., 6., 7., 8., 9.], [10., 11., 12., 13., 14.], [15., 16., 17., 18., 19.]]) Coordinates: * x (x) int64 0 1 2 3 * y (y) int64 0 1 2 3 4

Output of xr.show_versions()

• [x] Closes #2619
• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Fix using MultiIndex.remove_unused_levels()

• [x] Closes #2468
• [x] Tests added
• [N.A.] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

A fix for #2468.

• [x] Closes #2428

I installed pep8speaks as suggested in #2428. It looks they do not need a yml file, but it may be safer to add this (just renamed from .stickler.yml)

• [x] Closes #2440
• [x] Tests added
• [x] Tests passed
• [x] Fully documented, including whats-new.rst for all changes

It looks that I wrongly remove ddof option for nanstd in #2236. This PR fixes this.

• [x] Closes #2445
• [x] Tests added
• [x] Tests passed
• [x] Fully documented, including whats-new.rst for all changes

It is a regression after #2360.

Copied from issue#2236 comments, by @st-bender

Hi, just to let you know that .std() does not accept the ddof keyword anymore (it worked in 0.10.8) Should I open a new bugreport?

Edit: It fails with:

```python ~/Work/miniconda3/envs/stats/lib/python3.6/site-packages/xarray/core/duck_array_ops.py in f(values, axis, skipna, kwargs) 234 235 try: --> 236 return func(values, axis=axis, kwargs) 237 except AttributeError: 238 if isinstance(values, dask_array_type):

TypeError: nanstd() got an unexpected keyword argument 'ddof' ```

From #2356

APIs for groupby, groupby_bins, resample, rolling are different, especially for multi-dimensional array.

```python import numpy as np import xarray as xr import pandas as pd

time = pd.date_range('2000-01-01', freq='6H', periods=365 * 4) ds = xr.Dataset({'foo': (('time', 'x'), np.random.randn(365 * 4, 5)), 'time': time, 'x': [0, 1, 2, 1, 0]})

ds.rolling(time=2).mean() # result dims : ('time', 'x') ds.resample(time='M').mean() # result dims : ('time', 'x') ds['foo'].resample(time='M').mean() # result dims : ('time', ) maybe a bug #2362 ds.groupby('time.month').mean() # result dims : ('month', ) ds.groupby_bins('time', 3).mean() # result dims : ('time_bins', ) ```

• In rolling and resample(for Dataset), reduction without argument is carried out along grouped dimension
• In rolling, reduction along other dimesnion is not possible
• In groupby and groupby_bins, reduction is applied to the grouped objects and if without argument, it reduces alongall the dimensions of each grouped object.

I think rollings API is most clean, but I am not sure it is worth to change these APIs.

The possible options would be 1. Change APIs of groupby and groupby_bins so that they share similar API with rolling. 2. Document clearly how to perform resample or groupby with multidimensional arrays.

• [ ] Closes #xxxx
• [x] Tests added
• [x] Tests passed
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Started to fix #2363. Now warns a futurewarning in groupby if default reduction dimension is not specified. As a side effect, I added xarray.ALL_DIMS. With dim=ALL_DIMS always reduces along all the dimensions.

• [x] Closes #2230
• [x] Tests added
• [x] Tests passed
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

In #2230, the addition of min_count keywords for our reduction methods was discussed, but our duck_array_ops module is becoming messy (mainly due to nan-aggregation methods for dask, bottleneck and numpy) and it looks a little hard to maintain them.

I tried to refactor them by moving nan-aggregation methods to nanops module.

I think I still need to take care of more edge cases, but I appreciate any comment for the current implementation.

Note: In my implementation, bottleneck is not used when skipna=False. bottleneck would be advantageous when skipna=True as numpy needs to copy the entire array once, but I think numpy's method is still OK if skipna=False.

• [x] Closes #1332
• [x] Tests added
• [x] Tests passed
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Added xr.gradient, xr.DataArray.gradient, and xr.Dataset.gradient according to #1332.

• [x] Closes #2179
• [x] Tests passed
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

• [x] Tests passed (for all non-documentation changes)

Following to #2174

In some methods, consistency of the dictionary arguments and keyword arguments are checked twice in Dataset and Variable. Can we change the API of Variable so that it does not take kwargs-type argument for dimension names?

• [x] Closes #2299
• [x] Tests added
• [x] Tests passed
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API.

Now it raises an Error to assign new dimension with the name conflicting with an existing level name. Therefore, it is not allowed ```python b = xr.Dataset(coords={'dim0': ['a', 'b'], 'dim1': [0, 1]}) b = b.stack(dim_stacked=['dim0', 'dim1'])

This should raise an errors even though its length is consistent with b['dim0']

b['c'] = (('dim0',), [10, 11, 12, 13])

This is OK

b['c'] = (('dim_stacked',), [10, 11, 12, 13]) ```

I just mark some to_cdms2 tests xfail. See #2332 for the details. It is a temporal workaround and we may need to keep #2332 open until it is solved.

Recently, dask renamed dask.ghost to dask.overlap. We use them around rolling. For the patch, see #2349 .

BTW, there is another faling test in python=2.7 dev, claiming that

iris and pynio which gives various errors of arrays not being equal for test_to_and_from_cdms2_sgrid and test_to_and_from_cdms2_ugrid

Is anyone working on this? If not, I think we can temporally skip these tests for python 2.7.

Dask renamed dask.ghost -> dask.overlap in dask/dask#3830. This PR follows up this.

• [x] Closes #2341
• [x] Tests added
• [x] Tests passed
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Now raises a ValueError when the size of input_core_dims is inconsistent with number of argument.

From SO

In the following script, the second argument is silently neglected, python da = xr.DataArray(np.random.randn(4, 3), coords={'x': [5, 7, 9, 11]}, dims=('x', 'y')) xr.apply_ufunc(np.gradient, da, da.coords['x'].values, kwargs={'axis': -1}, input_core_dims=[['x']], output_core_dims=[['x']], output_dtypes=[da.dtype]) This is because we need to the same number of input_core_dims to the number of arguments,

https://github.com/pydata/xarray/blob/56381ef444c5e699443e8b4e08611060ad5c9507/xarray/core/computation.py#L535-L538

The correct scipt might be input_core_dims=[['x']] -> input_core_dims=[['x'], []].

I think we can raise a more friendly error if the size of input_core_dims are wrong.

EDIT: Or we can automatically insert an empty tuple or None for non-xarray object? input_core_dims for non-xarray object sounds a little strange.

Trivial changes to pass local flake8 tests.

I merged #2312 without making sure the building test passing, but there was a typo. Ths PR fixes it.

Some tests related to dask/distributed are failing in travis. They are raising a TypeError: can't pickle thread.lock objects.

Could anyone help to look inside?

See the travis's log for the current master: https://travis-ci.org/pydata/xarray/builds/392530577

• [x] Closes #1831, #1142, #819
• [x] Tests added
• [x] Tests passed
• [x] Passes git diff upstream/master **/*py | flake8 --diff
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

I started to work for refactoring rollings. As suggested in #1831 comment, I implemented rolling_window methods based on as_strided.

I got more than 1,000 times speed up! yey!

python In [1]: import numpy as np ...: import xarray as xr ...: ...: da = xr.DataArray(np.random.randn(10000, 3), dims=['x', 'y']) with the master python %timeit da.rolling(x=5).reduce(np.mean) 1 loop, best of 3: 9.68 s per loop with the current implementation python %timeit da.rolling(x=5).reduce(np.mean) 100 loops, best of 3: 5.29 ms per loop and with the bottleneck python %timeit da.rolling(x=5).mean() 100 loops, best of 3: 2.62 ms per loop

My current concerns are + Can we expose the new rolling_window method of DataArray and Dataset to the public? I think this method itself is useful for many usecases, such as short-term-FFT and convolution. This also gives more flexible rolling operation, such as windowed moving average, strided rolling, and ND-rolling.

• Is there any dask's equivalence to numpy's as_strided? Currently, I just use a slice->concatenate path, but I don't think it is very efficient. (Is it already efficient, as dask utilizes out-of-core computation?)

Any thoughts are welcome.

There is an error on rtd. http://xarray.pydata.org/en/stable/plotting.html#id4

• [x] Closes #2218
• [x] Tests added
• [x] Tests passed
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API.

This adds interp_like, that behaves like reindex_like but using interpolation.

Just as a reminder of the remaining extension of #2104 .

We might add interp_like that behaves like reindex_like but using interp().

• [x] Closes #2079 (remove if there is no corresponding issue, which should only be the case for minor changes)
• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

I started working to add interpolate_at to xarray, as discussed in issue #2079 (but without caching).

I think I need to take care of more edge cases, but before finishing up this PR, I want to discuss what the best API is.

I would like to this method working similar to isel, which may support vectorized interpolation. Currently, this works as follwos

```python In [1]: import numpy as np ...: import xarray as xr ...: ...: da = xr.DataArray([0, 0.1, 0.2, 0.1], dims='x', coords={'x': [0, 1, 2, 3]}) ...:

In [2]: # simple linear interpolation ...: da.interpolate_at(x=[0.5, 1.5]) ...: Out[2]: <xarray.DataArray (x: 2)> array([0.05, 0.15]) Coordinates: * x (x) float64 0.5 1.5

In [3]: # with cubic spline interpolation ...: da.interpolate_at(x=[0.5, 1.5], method='cubic') ...: Out[3]: <xarray.DataArray (x: 2)> array([0.0375, 0.1625]) Coordinates: * x (x) float64 0.5 1.5

In [4]: # interpolation at one single position ...: da.interpolate_at(x=0.5) ...: Out[4]: <xarray.DataArray ()> array(0.05) Coordinates: x float64 0.5

In [5]: # interpolation with broadcasting ...: da.interpolate_at(x=xr.DataArray([[0.5, 1.0], [1.5, 2.0]], dims=['y', 'z'])) ...: Out[5]: <xarray.DataArray (y: 2, z: 2)> array([[0.05, 0.1 ], [0.15, 0.2 ]]) Coordinates: x (y, z) float64 0.5 1.0 1.5 2.0 Dimensions without coordinates: y, z

In [6]: da = xr.DataArray([[0, 0.1, 0.2], [1.0, 1.1, 1.2]], ...: dims=['x', 'y'], ...: coords={'x': [0, 1], 'y': [0, 10, 20]}) ...:

In [7]: # multidimensional interpolation ...: da.interpolate_at(x=[0.5, 1.5], y=[5, 15]) ...: Out[7]: <xarray.DataArray (x: 2, y: 2)> array([[0.55, 0.65], [ nan, nan]]) Coordinates: * x (x) float64 0.5 1.5 * y (y) int64 5 15

In [8]: # multidimensional interpolation with broadcasting ...: da.interpolate_at(x=xr.DataArray([0.5, 1.5], dims='z'), ...: y=xr.DataArray([5, 15], dims='z')) ...: Out[8]: <xarray.DataArray (z: 2)> array([0.55, nan]) Coordinates: x (z) float64 0.5 1.5 y (z) int64 5 15 Dimensions without coordinates: z ```

Design question

1. How many interpolate methods should we support? Currently, I only implemented scipy.interpolate.interp1d for 1dimensional interpolation and scipy.interpolate.RegularGridInterpolator for multidimensional interpolation. I think 90% usecases are linear, but there are more methods in scipy.

2. How do we handle nan? Currently this raises ValueError if nan is present. It may be possible to carry out the interpolation with skipping nan, but in this case the performance would be significantly drops because it cannot be vectorized.

3. Do we support interpolation along dimension without coordinate? In that case, do we attach new coordinate to the object?

4. How should we do if new coordinate has the dimensional coordinate for the dimension to be interpolated? e.g. in the following case, python da = xr.DataArray([0, 0.1, 0.2, 0.1], dims='x', coords={'x': [0, 1, 2, 3]}) rslt = da.interpolate_at(x=xr.DataArray([0.5, 1.5], dims=['x'], coords={'x': [1, 3]}) what would be rslt['x']?

I appreciate any comments.

I noticed an example I added to doc in #2104 consumes more than 1 GB memory, and it results in the failing in readthedocs build.

This PR changes this to a much lighter example.

• [x] Closes #1932
• [x] Tests added
• [x] Tests passed
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

This precision drop was caused when converting pd.Timestamp to np.array python In [7]: ts = pd.Timestamp(np.datetime64('2018-02-12 06:59:59.999986560')) In [11]: np.asarray(ts, 'datetime64[ns]') Out[11]: array('2018-02-12T06:59:59.999986000', dtype='datetime64[ns]')

We need to call to_datetime64 explicitly.

• [x] Closes #1897
• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

I tried to support lazy vectorised indexing inspired by #1897. More tests would be necessary but I want to decide whether it is worth to continue.

My current implementation is + For outer/basic indexers, we combine successive indexers (as we are doing now). + For vectorised indexers, we just store them as is and index sequentially when the evaluation.

The implementation was simpler than I thought, but it has a clear limitation. It requires to load array before the vectorised indexing (I mean, the evaluation time). If we make a vectorised indexing for a large array, the performance significantly drops and it is not noticeable until the evaluation time.

I appreciate any suggestions.

• [x] Related with #2203
• [x] Tests added
• [x] Tests passed
• [x] Fully documented

Related with #2203, I think it is better if xr.DataArray.dot() is working even with older dask, at least in the simpler case (as this is a very primary operation).

The cost is a slight complication of the code. Any comments are welcome.

Code Sample, a copy-pastable example if possible

from #2180 , comment

python a = Dataset({ 'x': [10, 20], 'd1': ('x', [100, 200]), 'd2': ('x', [300, 400]) }) b = Dataset({ 'x': [15], 'd1': ('x', [500]), }) a.update(b) python Dataset({ 'x': [10, 20, 15], 'd1': ('x', [nan, nan, 500]), 'd2': ('x', [300, 400, nan]) })

In the above, with anything but an outer join you're destroying d2 - which doesn't even exist in the rhs dataset! A sane, desirable outcome should be

Problem description

Alignment should work

Expected Output

python Dataset({ 'x': [10, 20, 15], 'd1': ('x', [100, 200, 500]), 'd2': ('x', [300, 400, nan]) })

An example of weighted rolling mean in doc is actually weighted rolling sum. It is a little bit misleading SO, so I propose to change

weighted rolling mean -> weighted rolling sum

• [x] Closes #2112
• [x] Tests added
• [x] Tests passed
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API.

Now uses dataset_merge_method when a new coordinate is assigned to a xr.DataArray

Code Sample, a copy-pastable example if possible

I think we can raise an Error if the newly assigned coordinate to a DataArray has an invalid shape.

python In [1]: import xarray as xr ...: ...: da = xr.DataArray([0, 1, 2], dims='x') ...: da['x'] = [0, 1, 2, 3] # no error ...: da ...: Out[1]: <xarray.DataArray (x: 3)> array([0, 1, 2]) Coordinates: * x (x) int64 0 1 2 3

Problem description

It is more user-friendly if we make some sanity checks when a new coordinate is assigned to a xr.DataArray.

Dataset raises an appropriate error, python In [2]: ds = xr.Dataset({'da': ('x', [0, 1, 2])}) ...: ds['x'] = [0, 1, 2, 3] # -> raises ValueError

Expected Output

ValueError

• [ ] Closes #N.A.
• [x] Tests added (for all bug fixes or enhancements)
• [ ] Tests N.A.
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

I noticed rolling with dask array and with bottleneck can be slightly improved by reducing the padding depth in da.ghost.ghost(a, depth=depth, boundary=boundary).

@jhamman , can you kindly review this?

• [x] Closes #2099
• [x] Tests added
• [x] Tests passed

A quick fix for #2099

• [x] Closes #2113
• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Two bugs were found and fixed.

1. rolling a dask-array with center=True and bottleneck
2. rolling an integer dask-array with bottleneck

• [x] Closes #2114
• [x] Tests added
• [x] Tests passed
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Fixes 2114.

Code Sample, a copy-pastable example if possible

keep_attrs=True works nice for xr.DataArray, but it is neglected for xr.Variable

```python In [2]: import numpy as np In [3]: import xarray as xr In [4]: da = xr.DataArray([0, 1, 2], dims='x', attrs={'foo': 'var'}) In [5]: func = lambda x: x*2 In [6]: xr.apply_ufunc(func, da, keep_attrs=True, input_core_dims=[['x']], outpu ...: t_core_dims=[['z']]) Out[6]: <xarray.DataArray (z: 3)> # attrs are tracked for xr.DataArray array([0, 2, 4]) Dimensions without coordinates: z Attributes: foo: var

In [7]: xr.apply_ufunc(func, da.variable, keep_attrs=True, input_core_dims=[['x' ...: ]], output_core_dims=[['z']]) Out[7]: <xarray.Variable (z: 3)> # attrs are dropped array([0, 2, 4]) ```

Problem description

keep_attrs=True should work also with xr.Variable

Expected Output

python <xarray.Variable (z: 3)> # attrs are dropped array([0, 2, 4]) Attributes: foo: var

Code Sample, a copy-pastable example if possible

I noticed a bug introduced by #2087 (my PR)

```python import xarray as xr

ds = xr.Dataset({'var': ('x', [1, 2, 3])}, coords={'x': [0, 1, 2], 'z1': ('x', [1, 2, 3]), 'z2': ('x', [1, 2, 3])})

ds['var'] = ds['var'] * 2 ``` It claims a ValueError.

Problem description

https://github.com/pydata/xarray/blob/0cc64a08c672e6361d05acea3fea9f34308b62ed/xarray/core/merge.py#L564

Here should be python other[k] = obj.drop(coord_names) not python other[k] = obj.drop(*coord_names)

• [x] Closes #2068
• [x] Tests added
• [x] Tests passed
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

After this, when assigning a dataarray to a dataset, non-dimensional and conflicted coordinates of the dataarray are dropped.

example ``` In [2]: ds = xr.Dataset({'da': ('x', [0, 1, 2])}, ...: coords={'y': (('x',), [0.1, 0.2, 0.3])}) ...: ds ...: Out[2]: <xarray.Dataset> Dimensions: (x: 3) Coordinates: y (x) float64 0.1 0.2 0.3 Dimensions without coordinates: x Data variables: da (x) int64 0 1 2

In [3]: other = ds['da'] ...: other['y'] = 'x', [0, 1, 2] # conflicted non-dimensional coordinate ...: ds['da'] = other ...: ds ...: Out[3]: <xarray.Dataset> Dimensions: (x: 3) Coordinates: y (x) float64 0.1 0.2 0.3 # 'y' is not overwritten Dimensions without coordinates: x Data variables: da (x) int64 0 1 2 ```

• [x] Closes #1912

The removal of flake8 from travis would increase the clearer separation between style-issue and test failure.

Code Sample, a copy-pastable example if possible

apply_ufunc can generate an invalid object if the size of the array is changed by ufunc,

python In [1]: import numpy as np ...: import xarray as xr ...: ...: da = xr.DataArray([1, 2, 3], dims=['x'], coords={'x': [0, 0.5, 1]}) ...: ...: def tile(x): ...: return np.tile(da, [2]) ...: ...: tiled = xr.apply_ufunc(tile, da, input_core_dims=[['x']], ...: output_core_dims=[['x']]) ...: tiled ...: Out[1]: <xarray.DataArray (x: 6)> array([1, 2, 3, 1, 2, 3]) Coordinates: * x (x) float64 0.0 0.5 1.0

In the above example, tiled.shape = (6, ) but tiled['x'].shape = (3,). I think we need a sanity check and drop coordinate if necessary.

Problem description

Any of our function should not generate invalid xarray objects.

Expected Output

python Out[1]: <xarray.DataArray (x: 6)> array([1, 2, 3, 1, 2, 3]) Dimensions without coordinates: x

or raise an Error.

Output of xr.show_versions()

Code Sample, a copy-pastable example if possible

```python

da = xr.DataArray([0, 1, 2], dims='x') 0 in da ( warning omitted ) False 0 in da.values True ```

Problem description

__contains__ should work as np.ndarray does.

Expected Output

```python

0 in da True ```

• [x] Closes #1993
• [x] Tests passed
• [x] Fully documented, including whats-new.rst for all changes.

Quick fix of #1993.

With this fix, the script shown in #1993 runs Bottleneck: 0.08317923545837402 s Pandas: 1.3338768482208252 s Xarray: 1.1349339485168457 s

I found the following page fails http://xarray.pydata.org/en/stable/examples/weather-data.html

• [x] Closes #1951
• [x] Tests added
• [x] Tests passed (for all non-documentation changes)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

Currently, lazy-einsum for dask is not yet working.

@shoyer I think apply_ufunc supports lazy computation, but I did not yet figure out how to do this. Can you give me a help?

Code Sample, a copy-pastable example if possible

I sometimes want to make more flexible dot product of two data arrays, where we sum up along a part of common dimensions.

```python

Your code here

da_vals = np.arange(6 * 5 * 4).reshape((6, 5, 4)) da = DataArray(da_vals, dims=['x', 'y', 'z']) dm_vals = np.arange(6 * 4).reshape((6, 4)) dm = DataArray(dm_vals, dims=['x', 'z'])

I want something like this

da.dot(dm, 'z') # -> dimensions of the output array: ['x', 'y'] ```

It's an intermediate path of np.matmul and np.tensordot.

Is this feature sufficiently universal?

EDIT: I just noticed dask does not have einsum yet (dask/dask#732). We maybe need to wait or decide to support only numpy arrays.

Problem description

Tests are failing due to the release of zarr 2.2.0

Travis's log https://travis-ci.org/pydata/xarray/jobs/351566529

I noticed the failing in rtd http://xarray.pydata.org/en/stable/computation.html#missing-values is because it still uses numpy=1.11 which does not support signature argument for np.vectorize.

This can be easily fixed (just bumping up numpy's version on rtd), but as our minimum requirement is numpy==1.11, we may need to take care of this in xr.apply_ufunc.

• [x] Partly closes #1944

I noticed this is due to the use of old numpy on rtd.

xref #1956

• [x] Closes #1944

Code Sample, a copy-pastable example if possible

```python In [1]: import numpy as np ...: import xarray as xr ...: ...: da = xr.DataArray(np.random.randn(1000, 100), dims=['x', 'y'], ...: coords={'x': np.arange(1000)}) ...:

In [2]: %%timeit ...: da.rolling(x=10).reduce(np.sum) ...: 2.04 s ± 8.25 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) ```

Problem description

In DataArray.rolling, we index by .isel method for every window, constructing huge number of xr.DataArray instances. This is very inefficient.

Of course, we can use bottleneck methods if available, but this provides only a limited functions. (This also limits possible extensions of rolling, such as ND-rolling (#819), window type (#1142), strides (#819).)

I am wondering if we could skip any sanity checks in our DataArray.isel -> Variable.isel path in indexing. Or can we directly construct a single large DataArray instead of a lot of small DataArrays?

Typo in url.

From gitter cc. @davidh-ssec

python In [1]: import numpy as np ...: import xarray as xr ...: ...: da = xr.DataArray(np.arange(100), dims='x').chunk({'x': 10}) ...: da.isnull() ...: Out[1]: <xarray.DataArray (x: 100)> array([False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False], dtype=bool) Dimensions without coordinates: x

Problem description

DataArray.isnull() automatically computes dask arrays.

Expected Output

python Out[2]: <xarray.DataArray (x: 100)> dask.array<shape=(100,), dtype=bool, chunksize=(10,)> Dimensions without coordinates: x

Cause

https://github.com/pydata/xarray/blob/697cc74b9af5fbfedadd54fd07019ce7684553ec/xarray/core/ops.py#L322-L324

Here, getattr(pd, name) should be getattr(duck_array_ops, name).

• [x] Closes #1921
• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

Now Variable accepts 0d-np.ndarray indexer.

```python In [1]: import xarray as xr ...: import numpy as np ...: ...: a = np.arange(10) ...: a[np.array(0)] ...: Out[1]: 0

In [2]: da = xr.DataArray(a, dims='x') ...: da[np.array(0)] ...:

TypeError Traceback (most recent call last) <ipython-input-2-d30fdfc612ec> in <module>() 1 da = xr.DataArray(a, dims='x') ----> 2 da[np.array(0)]

/home/keisukefujii/Dropbox/projects/xarray.git/xarray/core/dataarray.pyc in getitem(self, key) 478 else: 479 # xarray-style array indexing --> 480 return self.isel(**self._item_key_to_dict(key)) 481 482 def setitem(self, key, value):

/home/keisukefujii/Dropbox/projects/xarray.git/xarray/core/dataarray.pyc in isel(self, drop, indexers) 759 DataArray.sel 760 """ --> 761 ds = self._to_temp_dataset().isel(drop=drop, indexers) 762 return self._from_temp_dataset(ds) 763

/home/keisukefujii/Dropbox/projects/xarray.git/xarray/core/dataset.py in isel(self, drop, indexers) 1390 for name, var in iteritems(self._variables): 1391 var_indexers = {k: v for k, v in indexers_list if k in var.dims} -> 1392 new_var = var.isel(var_indexers) 1393 if not (drop and name in var_indexers): 1394 variables[name] = new_var

/home/keisukefujii/Dropbox/projects/xarray.git/xarray/core/variable.pyc in isel(self, **indexers) 851 if dim in indexers: 852 key[i] = indexers[dim] --> 853 return self[tuple(key)] 854 855 def squeeze(self, dim=None):

/home/keisukefujii/Dropbox/projects/xarray.git/xarray/core/variable.pyc in getitem(self, key) 619 array x.values directly. 620 """ --> 621 dims, indexer, new_order = self._broadcast_indexes(key) 622 data = as_indexable(self._data)[indexer] 623 if new_order:

/home/keisukefujii/Dropbox/projects/xarray.git/xarray/core/variable.pyc in _broadcast_indexes(self, key) 477 # key can be mapped as an OuterIndexer. 478 if all(not isinstance(k, Variable) for k in key): --> 479 return self._broadcast_indexes_outer(key) 480 481 # If all key is 1-dimensional and there are no duplicate labels,

/home/keisukefujii/Dropbox/projects/xarray.git/xarray/core/variable.pyc in _broadcast_indexes_outer(self, key) 542 new_key.append(k) 543 --> 544 return dims, OuterIndexer(tuple(new_key)), None 545 546 def _nonzero(self):

/home/keisukefujii/Dropbox/projects/xarray.git/xarray/core/indexing.py in init(self, key) 368 raise TypeError('invalid indexer array for {}, must have ' 369 'exactly 1 dimension: ' --> 370 .format(type(self).name, k)) 371 k = np.asarray(k, dtype=np.int64) 372 else:

TypeError: invalid indexer array for OuterIndexer, must have exactly 1 dimension: ```

Indexing by a 0d-array should be identical to the indexing by a scalar.

• [x] Closes #1866
• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

I am working to add aggregation ops for object-typed arrays, which may make #1837 cleaner. I added some tests but maybe not sufficient. Any other cases which should be considered? e.g. [True, 3.0, np.nan] etc...

This issue arises in #1837 comment, where we need to make a summation of object-dtype array, such as python xr.DataArray(np.array([True, True, False, np.nan], dtype=object), dims='x').sum('x', skipna=True) Currently, it raises NotImplementedError.

pandas support this by having their own nan-aggregation methods.