• [x] Closes #1444, closes #1436
• [x] Tests added / passed
• [x] Passes git diff master | flake8 --diff
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

xref https://github.com/pydata/xarray/issues/974#issuecomment-313977794

• [x] Closes #1257
• [x] Tests added / passed
• [x] Passes git diff upstream/master | flake8 --diff
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

This is a very bare bones addition of the asv benchmarking tool to xarray. I have added four very rudimentary benchmarks in the dataset_io.py module.

Usage of asv is pretty straightforward but I'll outline the steps for those who want to try this out: cd xarray conda install asv -c conda-forge asv run # this will install some conda environments in ./.asv/envs asv publish # this collates the results asv preview # this will launch a web server so you can visually compare the tests

Before I go any further, I want to get some input from @pydata/xarray on what we want to see in this PR. In previous projects, I have found designing tests after the fact can end up being fairly arbitrary and I want to avoid that if at all possible. I'm guessing that we will want to focus our efforts for now on I/O and dask related performance but how we do that is up for discussion.

cc @shoyer, @rabernat, @MaximilianR, @Zac-HD

As of xarray-0.10-rc1:

Works: ``` import xarray import scipy.stats a = xarray.DataArray([1,2], dims=['x'])

xarray.apply_ufunc(scipy.stats.norm.cdf, a, 0, 1)

<xarray.DataArray (x: 2)> array([ 0.841345, 0.97725 ]) Dimensions without coordinates: x ```

Broken: ``` xarray.apply_ufunc( scipy.stats.norm.cdf, a.chunk(), 0, 1, dask='parallelized', output_dtypes=[a.dtype] ).compute()

IndexError Traceback (most recent call last) <ipython-input-35-1d4025e1ebdb> in <module>() ----> 1 xarray.apply_ufunc(scipy.stats.norm.cdf, a.chunk(), 0, 1, dask='parallelized', output_dtypes=[a.dtype]).compute()

~/anaconda3/lib/python3.6/site-packages/xarray/core/computation.py in apply_ufunc(func, args, kwargs) 913 join=join, 914 exclude_dims=exclude_dims, --> 915 keep_attrs=keep_attrs) 916 elif any(isinstance(a, Variable) for a in args): 917 return variables_ufunc(args)

~/anaconda3/lib/python3.6/site-packages/xarray/core/computation.py in apply_dataarray_ufunc(func, args, kwargs) 210 211 data_vars = [getattr(a, 'variable', a) for a in args] --> 212 result_var = func(data_vars) 213 214 if signature.num_outputs > 1:

~/anaconda3/lib/python3.6/site-packages/xarray/core/computation.py in apply_variable_ufunc(func, args, kwargs) 561 raise ValueError('unknown setting for dask array handling in ' 562 'apply_ufunc: {}'.format(dask)) --> 563 result_data = func(input_data) 564 565 if signature.num_outputs > 1:

~/anaconda3/lib/python3.6/site-packages/xarray/core/computation.py in <lambda>(arrays) 555 func = lambda arrays: _apply_with_dask_atop( 556 numpy_func, arrays, input_dims, output_dims, signature, --> 557 output_dtypes, output_sizes) 558 elif dask == 'allowed': 559 pass

~/anaconda3/lib/python3.6/site-packages/xarray/core/computation.py in _apply_with_dask_atop(func, args, input_dims, output_dims, signature, output_dtypes, output_sizes) 624 for element in (arg, dims[-getattr(arg, 'ndim', 0):])] 625 return da.atop(func, out_ind, *atop_args, dtype=dtype, concatenate=True, --> 626 new_axes=output_sizes) 627 628

~/anaconda3/lib/python3.6/site-packages/dask/array/core.py in atop(func, out_ind, args, kwargs) 2231 raise ValueError("Must specify dtype of output array") 2232 -> 2233 chunkss, arrays = unify_chunks(args) 2234 for k, v in new_axes.items(): 2235 chunkss[k] = (v,)

~/anaconda3/lib/python3.6/site-packages/dask/array/core.py in unify_chunks(args, *kwargs) 2117 chunks = tuple(chunkss[j] if a.shape[n] > 1 else a.shape[n] 2118 if not np.isnan(sum(chunkss[j])) else None -> 2119 for n, j in enumerate(i)) 2120 if chunks != a.chunks and all(a.chunks): 2121 arrays.append(a.rechunk(chunks))

~/anaconda3/lib/python3.6/site-packages/dask/array/core.py in <genexpr>(.0) 2117 chunks = tuple(chunkss[j] if a.shape[n] > 1 else a.shape[n] 2118 if not np.isnan(sum(chunkss[j])) else None -> 2119 for n, j in enumerate(i)) 2120 if chunks != a.chunks and all(a.chunks): 2121 arrays.append(a.rechunk(chunks))

IndexError: tuple index out of range ```

Workaround: ``` xarray.apply_ufunc( scipy.stats.norm.cdf, a, kwargs={'loc': 0, 'scale': 1}, dask='parallelized', output_dtypes=[a.dtype]).compute()

<xarray.DataArray (x: 2)> array([ 0.841345, 0.97725 ]) Dimensions without coordinates: x ```

Code Sample, a copy-pastable example if possible

```python import numpy as np import xarray as xr

a = np.random.randn(4, 3) a[1, 1] = np.NaN da = xr.DataArray(a, dims=('y', 'x'), coords={'y':np.arange(4), 'x':np.arange(3)}) da.to_netcdf('test.nc')

with xr.open_dataarray('test.nc') as da: da.dropna(dim='x', how='any')

ValueError Traceback (most recent call last) <ipython-input-37-8d137cf3a813> in <module>() 8 9 with xr.open_dataarray('test.nc') as da: ---> 10 da.dropna(dim='x', how='any')

~/.pyvirtualenvs/py3/lib/python3.5/site-packages/xarray/core/dataarray.py in dropna(self, dim, how, thresh) 1158 DataArray 1159 """ -> 1160 ds = self._to_temp_dataset().dropna(dim, how=how, thresh=thresh) 1161 return self._from_temp_dataset(ds) 1162

~/.pyvirtualenvs/py3/lib/python3.5/site-packages/xarray/core/dataset.py in dropna(self, dim, how, thresh, subset) 2292 raise TypeError('must specify how or thresh') 2293 -> 2294 return self.isel(**{dim: mask}) 2295 2296 def fillna(self, value):

~/.pyvirtualenvs/py3/lib/python3.5/site-packages/xarray/core/dataset.py in isel(self, drop, **indexers) 1291 coord_names = set(variables).intersection(self._coord_names) 1292 selected = self._replace_vars_and_dims(variables, -> 1293 coord_names=coord_names) 1294 1295 # Extract coordinates from indexers

~/.pyvirtualenvs/py3/lib/python3.5/site-packages/xarray/core/dataset.py in _replace_vars_and_dims(self, variables, coord_names, dims, attrs, inplace) 598 """ 599 if dims is None: --> 600 dims = calculate_dimensions(variables) 601 if inplace: 602 self._dims = dims

~/.pyvirtualenvs/py3/lib/python3.5/site-packages/xarray/core/dataset.py in calculate_dimensions(variables) 111 raise ValueError('conflicting sizes for dimension %r: ' 112 'length %s on %r and length %s on %r' % --> 113 (dim, size, k, dims[dim], last_used[dim])) 114 return dims 115

ValueError: conflicting sizes for dimension 'y': length 2 on <this-array> and length 4 on 'y' ```

Problem description

See above. Note that the code runs when: - data is previously read into memory with load() - the DataArray is stored without coordinates (this is strange) - dropna is applied to 'y' instead of 'x'

Expected Output

This used to work in v0.9.6

Output of xr.show_versions()

I think this is a regression in the current 0.10.0rc1:

Code Sample

```python import xarray as xr ds = xr.open_dataset('cesm_data.nc', decode_cf=False) ds.temp.isel(time=ds.time < 274383) # throws an error

NotImplementedError Traceback (most recent call last) <ipython-input-18-a5c4179cd02d> in <module>() ----> 1 ds.temp.isel(time=ds.time < 274383)

~/.pyvirtualenvs/py3/lib/python3.5/site-packages/xarray/core/dataarray.py in isel(self, drop, indexers) 717 DataArray.sel 718 """ --> 719 ds = self._to_temp_dataset().isel(drop=drop, indexers) 720 return self._from_temp_dataset(ds) 721

~/.pyvirtualenvs/py3/lib/python3.5/site-packages/xarray/core/dataset.py in isel(self, drop, indexers) 1278 for name, var in iteritems(self._variables): 1279 var_indexers = {k: v for k, v in indexers_list if k in var.dims} -> 1280 new_var = var.isel(var_indexers) 1281 if not (drop and name in var_indexers): 1282 variables[name] = new_var

~/.pyvirtualenvs/py3/lib/python3.5/site-packages/xarray/core/variable.py in isel(self, **indexers) 771 if dim in indexers: 772 key[i] = indexers[dim] --> 773 return self[tuple(key)] 774 775 def squeeze(self, dim=None):

~/.pyvirtualenvs/py3/lib/python3.5/site-packages/xarray/core/variable.py in getitem(self, key) 595 """ 596 dims, index_tuple, new_order = self._broadcast_indexes(key) --> 597 data = self._indexable_data[index_tuple] 598 if new_order: 599 data = np.moveaxis(data, range(len(new_order)), new_order)

~/.pyvirtualenvs/py3/lib/python3.5/site-packages/xarray/core/indexing.py in getitem(self, key) 414 415 def getitem(self, key): --> 416 return type(self)(_wrap_numpy_scalars(self.array[key])) 417 418 def setitem(self, key, value):

~/.pyvirtualenvs/py3/lib/python3.5/site-packages/xarray/core/indexing.py in getitem(self, key) 394 395 def getitem(self, key): --> 396 return type(self)(_wrap_numpy_scalars(self.array[key])) 397 398 def setitem(self, key, value):

~/.pyvirtualenvs/py3/lib/python3.5/site-packages/xarray/core/indexing.py in getitem(self, key) 361 362 def getitem(self, key): --> 363 return type(self)(self.array, self._updated_key(key)) 364 365 def setitem(self, key, value):

~/.pyvirtualenvs/py3/lib/python3.5/site-packages/xarray/core/indexing.py in _updated_key(self, new_key) 336 raise NotImplementedError( 337 'Vectorized indexing for {} is not implemented. Load your ' --> 338 'data first with .load() or .compute().'.format(type(self))) 339 new_key = iter(expanded_indexer(new_key, self.ndim)) 340 key = []

NotImplementedError: Vectorized indexing for <class 'xarray.core.indexing.LazilyIndexedArray'> is not implemented. Load your data first with .load() or .compute().

``` Here is the file: cesm_data.nc.zip

Expected Output

This used to work in v0.9

Output of xr.show_versions()

• [x] Closes #1512
• [x] Tests passed
• [x] Passes git diff upstream/master | flake8 --diff
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Explicitly deprecated old pandas (< 0.18) and old numpy (< 1.11) supports. Some backported functions in npcompat are removed because numpy == 1.11 already has them.

Code Sample, a copy-pastable example if possible

```python

Your code here

import xarray as xr ds = xr.Dataset({'a': ('x', [0, 1, 2])}) ds. -> press 'Tab' ```

Problem description

IPython autocomplete raises a deprecation warning, introducing in #1643. ipython /home/keisukefujii/anaconda3/envs/tensorflow/lib/python3.5/site-packages/jedi/e. getattr(obj, name) /home/keisukefujii/anaconda3/envs/tensorflow/lib/python3.5/site-packages/jedi/e. obj = getattr(obj, name) In [3]: ds.

Expected Output

None

Output of xr.show_versions()

• [x] xref #986
• [x] Tests added / passed
• [x] Passes git diff upstream/master | flake8 --diff
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

This PR adds a new module level function xr.show_versions() and a new Github Issue Template to help with debugging user issues. Hopefully, we stop having to ask "what version of dask are you using?" Functionality is copied from pandas.

• [x] Closes #1093
• [x] Tests added / passed
• [x] Passes git diff upstream/master | flake8 --diff
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Working towards on a solution for #1462

Just some stub code for the moment.

Dask dataframes don't appear to support MultiIndex so not sure what to do about that.

apply_ufunc() does not meet all our needs for wrapping unlabeled array routines with xarray (see #1618 for a proposal for apply_raw()), but it should be useful for many advanced users and isn't doing much in its current state as non-public API. So I'd like to make it public for the xarray 0.10 release.

• [x] Closes #770
• [x] Passes git diff upstream/master | flake8 --diff
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API
• [x] Turn the example from #1517 into a separate docs page of some sort, and link to it from computation.rst and dask.rst.

@MaximilianR @rabernat @jhamman Review from any of you would be appreciated here!

Currently, we strip labels from DataArray arguments to [] / .loc[] / .sel()/ .isel(). But this will break when we finally add indexing with alignment (https://github.com/pydata/xarray/issues/974).

We could start raising deprecation warnings now so users can stop relying on this functionality that will change.

• [x] Closes #1521
• [x] Tests added / passed
• [x] Passes git diff upstream/master | flake8 --diff
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

This is a work-in-progress to resolve #1269.

• [x] Basic functionality
• [x] Cleanly deprecate old API
• [x] New test cases
• [x] Documentation / examples
• [x] "What's new"

Openly welcome feedback/critiques on how I approached this. Subclassing Data{Array/set}GroupBy may not be the best way, but it would be easy enough to re-write the necessary helper functions (just apply(), I think) so that we do not need to inherit form them directly. Additional issues I'm working to resolve:

• [x] I tried make sure that calls using the old API won't break by refactoring the old logic to _resample_immediately(). This may not be the best approach!
• [x] Similarly, I copied all the original test cases and added the suffix ..._old_api; these could trivially be placed into their related test cases for the new API.
• [x] BUG: keep_attrs is ignored when you call it on methods chained to Dataset.resample(). Oddly enough, if I hard-code keep_attrs=True inside reduce_array() in DatasetResample::reduce it works just fine. I haven't figured out where the kwarg is getting lost.
• [x] BUG: Some of the test cases (for instance, test_resample_old_vs_new_api) fail because the resampling by calling self.groupby_cls ends up not working - it crashes because the group sizes that get computed are not what it expects. Occurs with both new and old API

• [x] Fixes #1522
• [x] Tests added / passed
• [x] Passes git diff upstream/master | flake8 --diff
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Stop dataset data vars and non-index dataset/dataarray coords from being loaded by repr() and getattr(). The latter is particularly acute when working in Jupyter, which does a dozen or so getattr() when printing an object.

• [x] Closes #1120
• [x] Tests added / passed
• [x] Passes git diff upstream/master | flake8 --diff
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

We currently have 7 failing tests when run against the pandas development code (travis).

Question for @shoyer - can you take a look at these and see if we should try to get a fix in place prior to v.0.10.0? It looks like Pandas.0.21 is slated for release on Sept. 30.

Make the filename argument to NetCDF4DataStore polymorphic so that a Dataset can be passed in.

• [x] Closes #1459
• [x] Tests added / passed
• [x] Passes git diff upstream/master | flake8 --diff
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

1459 discussed adding an alternate constructor (i.e. a class method) to NetCDF4DataStore to allow this, which would be my preferred approach rather than making a filename polymorphic (via isinstance). Unfortunately, alternate constructors only work by taking one set of parameters (or setting defaults) and then passing them to the original constructor. Given that, there's no way to make an alternate constructor without also making the original constructor somehow aware of this functionality--or breaking backwards-compatibility. I'm open to suggestions to the contrary.

We need pandas >= 0.18 because dataframe.rolling is supported after 0.18. But requirements in our setup.py says we need pandas >= 0.15.

Additionally, I noticed that in travis's CONDA_ENV=py27-min setup, our unit tests run with pandas == 0.20, though it might be intended to run with pandas == 0.15.

By conda remove scipy, pandas.0.15 is removed. (Here is the travis log) ``` if [[ "$CONDA_ENV" == "py27-min" ]]; then conda remove scipy; fi Fetching package metadata ......... Solving package specifications: .

Package plan for package removal in environment /home/travis/miniconda/envs/test_env:

The following packages will be REMOVED:

pandas: 0.15.0-np19py27_0 defaults
scipy:  0.17.1-np19py27_1 defaults

then in `python setup.py install`, pandas==0.20.3 is installed. Searching for pandas>=0.15.0 Reading https://pypi.python.org/simple/pandas/ Downloading https://pypi.python.org/packages/ee/aa/90c06f249cf4408fa75135ad0df7d64c09cf74c9870733862491ed5f3a50/pandas-0.20.3.tar.gz#md5=4df858f28b4bf4fa07d9fbb7f2568173 Best match: pandas 0.20.3 ```

Example usage: ```python

a.where(a.x + a.y < 5, -1) <xarray.DataArray (x: 5, y: 5)> array([[ 0, 1, 2, 3, 4], [ 5, 6, 7, 8, -1], [10, 11, 12, -1, -1], [15, 16, -1, -1, -1], [20, -1, -1, -1, -1]]) Dimensions without coordinates: x, y ` - [x] Closes #576 - [x] Tests added / passed - [x] Passesgit diff upstream/master | flake8 --diff` - [x] Fully documented, includingwhats-new.rstfor all changes andapi.rst` for new API

CC @MaximilianR

• [x] Closes #1477
• [x] Tests added / passed
• [x] passes git diff upstream/master | flake8 --diff
• [x] added whats-new.rst entry.

This is a temporary work around for https://github.com/pandas-dev/pandas/issues/16896, which was introduced by pandas 0.20.3

We can safely revert it after the next pandas release.

• [x] Tests added / passed
• [x] Passes git diff upstream/master | flake8 --diff
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

~~This still probably should have a regression test.~~ Done

• [x] Closes #1444
• [x] Tests added / passed
• [x] Passes git diff upstream/master | flake8 --diff
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Solves this error:

TypeError: 'NotImplementedType' object is not callable

Adds the 'res', 'is_tiled', and 'transform' attributes to xarray's rasterio backend.

• [X] Closes #1456
• [X] Tests added / passed
• [X] Passes git diff upstream/master | flake8 --diff
• [X] Fully documented, including whats-new.rst for all changes and api.rst for new API

EDIT: fixed typo; 'tiled' attribute name updated to 'is_tiled'

Rasterio uses edge-based coordinates, which is a different convention from how xarray treats coordinates (based on description here: http://xarray.pydata.org/en/stable/plotting.html#coordinates). This PR centers them, offsetting by half of the resolution.

• [X] Tests added / passed
• [X] Passes git diff upstream/master | flake8 --diff ~- [ ] Closes #xxxx~
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

CCing @fmaussion since he may be interested.

I've spotted this wrong error message. In my opinion this is too small to justify an entry in whats-new.rst.