Importing version 0.11 of xarray fails if version 1.0.0 of Bottleneck is installed. Bottleneck seems to be an optional dependency of xarray. During runtime xarray replaces functions by their bottleneck versions if that is installed, but it does not check if the version of bottleneck that is installed is new enough to provide that function:

The getattr here fails with an AttributeError in this case:

https://github.com/pydata/xarray/blob/b46fcd656391d786b8d25b0615f6d4bd30b524b7/xarray/core/ops.py#L361-L365

AttributeError: 'module' object has no attribute 'move_argmax'

move_argmax was included into bottleneck in version 1.1.0, so if version 1.0 is installed this can't work.

I saw this on python2.7, but I don't think that should matter...

If we convert a pandas dataframe with a multiindex, slice it to remove some entries from the index, a converted DataArray still contains the removed items in the coordinates (although the values are NaN).

```python

We create an example dataframe

idx = pd.MultiIndex.from_product([list('abc'), list('xyz')]) df = pd.DataFrame(data={'col': np.random.randn(len(idx))}, index=idx) df.columns.name = 'cols' df.index.names = ['idx1', 'idx2'] df2 = df.loc[['a', 'b']] python

df2 does not contain c in the first level

df2 cols col idx1 idx2
a x -0.844476 y -0.845998 z 1.965143 b x -0.159293 y 0.188163 z -1.076204

It still shows up in the converted xarray though:

xr.DataArray(df2).unstack('dim_0') <xarray.DataArray (cols: 1, idx1: 3, idx2: 3)> array([[[-0.844476, -0.845998, 1.965143], [-0.159293, 0.188163, -1.076204], [ nan, nan, nan]]]) Coordinates: * cols (cols) object 'col' * idx1 (idx1) object 'a' 'b' 'c' * idx2 (idx2) object 'x' 'y' 'z' ```

If the original dataframe is very sparse, this can lead to gigantic unnecessary memory usage.

If we write a dask array that doesn't involve dask.delayed functions using ds.to_netcdf, there is only little overhead from pickle:

python vals = da.random.random(500, chunks=(1,)) ds = xr.Dataset({'vals': (['a'], vals)}) write = ds.to_netcdf('file2.nc', compute=False) %prun -stime -l10 write.compute()

``` 123410 function calls (104395 primitive calls) in 13.720 seconds

Ordered by: internal time List reduced from 203 to 10 due to restriction <10>

ncalls tottime percall cumtime percall filename:lineno(function) 8 10.032 1.254 10.032 1.254 {method 'acquire' of '_thread.lock' objects} 1001 2.939 0.003 2.950 0.003 {built-in method _pickle.dumps} 1001 0.614 0.001 3.569 0.004 pickle.py:30(dumps) 6504/1002 0.012 0.000 0.021 0.000 utils.py:803(convert) 11507/1002 0.010 0.000 0.019 0.000 utils_comm.py:144(unpack_remotedata) 6013 0.009 0.000 0.009 0.000 utils.py:767(tokey) 3002/1002 0.008 0.000 0.017 0.000 utils_comm.py:181(<listcomp>) 11512 0.007 0.000 0.008 0.000 core.py:26(istask) 1002 0.006 0.000 3.589 0.004 worker.py:788(dumps_task) 1 0.005 0.005 0.007 0.007 core.py:273(<dictcomp>) ```

But if we use results from dask.delayed, pickle takes up most of the time: ```python @dask.delayed def make_data(): return np.array(np.random.randn())

vals = da.stack([da.from_delayed(make_data(), (), np.float64) for _ in range(500)]) ds = xr.Dataset({'vals': (['a'], vals)}) write = ds.to_netcdf('file5.nc', compute=False) %prun -stime -l10 write.compute() ```

``` 115045243 function calls (104115443 primitive calls) in 67.240 seconds

Ordered by: internal time List reduced from 292 to 10 due to restriction <10>

ncalls tottime percall cumtime percall filename:lineno(function) 8120705/501 17.597 0.000 59.036 0.118 pickle.py:457(save) 2519027/501 7.581 0.000 59.032 0.118 pickle.py:723(save_tuple) 4 6.978 1.745 6.978 1.745 {method 'acquire' of '_thread.lock' objects} 3082150 5.362 0.000 8.748 0.000 pickle.py:413(memoize) 11474396 4.516 0.000 5.970 0.000 pickle.py:213(write) 8121206 4.186 0.000 5.202 0.000 pickle.py:200(commit_frame) 13747943 2.703 0.000 2.703 0.000 {method 'get' of 'dict' objects} 17057538 1.887 0.000 1.887 0.000 {built-in method builtins.id} 4568116 1.772 0.000 1.782 0.000 {built-in method _struct.pack} 2762513 1.613 0.000 2.826 0.000 pickle.py:448(get) ```

This additional pickle overhead does not happen if we compute the dataset without writing it to a file.

I am using dask.distributed. I haven't tested it with anything else.

Software versions

Dataset allows assignment of new variables with dimension names that are used in a MultiIndex, even if the lengths do not match the existing coordinate.

```python a = pd.DataFrame({'a': [1, 2], 'b': [3, 4]}).unstack('a') a.index.names = ['dim0', 'dim1'] a.index.name = 'stacked_dim'

b = xr.Dataset(coords={'dim0': ['a', 'b'], 'dim1': [0, 1]}) b = b.stack(dim_stacked=['dim0', 'dim1']) assert(len(b.dim0) == 4)

This should raise an errors because the length is != 4

b['c'] = (('dim0',), [10, 11]) b Instead, it reports `dim0` as a new dimension without coordinates: <xarray.Dataset> Dimensions: (dim0: 2, dim_stacked: 4) Coordinates: * dim_stacked (dim_stacked) MultiIndex - dim0 (dim_stacked) object 'a' 'a' 'b' 'b' - dim1 (dim_stacked) int64 0 1 0 1 Dimensions without coordinates: dim0 Data variables: c (dim0) int64 10 11 ```

Similar cases of coordinates that aren't used do raise an error: python ds = xr.Dataset() ds.coords['a'] = [1, 2, 3] ds = ds.sel(a=1) ds['b'] = (('a',), [1, 2]) ds

Output of xr.show_versions()