issues

Multi-dimensional grouped operations should be relatively straightforward -- the main complexity will be writing an N-dimensional concat that doesn't involve repetitively copying data.

The idea with group_over would be to support groupby operations that act on a single element from each of the given groups, rather than the unique values. For example, ds.group_over(['lat', 'lon']) would let you iterate over or apply to 2D slices of ds, no matter how many dimensions it has.

Roughly speaking (it's a little more complex for the case of non-dimension variables), ds.group_over(dims) would get translated into ds.groupby([d for d in ds.dims if d not in dims]).

Related: #266

This works with isel and sel (see below). There's no particular reason why it can't work with reindex and interp, too, though we would need to implement our own vectorized version of linear interpolation (should not be too bad, it's mostly a matter of indexing twice and calculating weights from the difference in coordinate values).

Apparently this is quite important for vertical regridding in weather/climate science (cc @rabernat, @nbren12 ) ``` In [35]: import xarray as xr

In [36]: import numpy as np

In [37]: data = xr.DataArray(np.arange(12).reshape((3, 4)), [('x', np.arange(3)), ('y', np.arange(4))])

In [38]: ind = xr.DataArray([[0, 2], [1, 0], [1, 2]], dims=['x', 'z'], coords={'x': [0, 1, 2]})

In [39]: data Out[39]: <xarray.DataArray (x: 3, y: 4)> array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) Coordinates: * x (x) int64 0 1 2 * y (y) int64 0 1 2 3

In [40]: ind Out[40]: <xarray.DataArray (x: 3, z: 2)> array([[0, 2], [1, 0], [1, 2]]) Coordinates: * x (x) int64 0 1 2 Dimensions without coordinates: z

In [41]: data.isel(y=ind) Out[41]: <xarray.DataArray (x: 3, z: 2)> array([[ 0, 2], [ 5, 4], [ 9, 10]]) Coordinates: * x (x) int64 0 1 2 y (x, z) int64 0 2 1 0 1 2 Dimensions without coordinates: z

In [42]: data.sel(y=ind) Out[42]: <xarray.DataArray (x: 3, z: 2)> array([[ 0, 2], [ 5, 4], [ 9, 10]]) Coordinates: * x (x) int64 0 1 2 y (x, z) int64 0 2 1 0 1 2 Dimensions without coordinates: z

In [43]: data.interp(y=ind)

ValueError Traceback (most recent call last) <ipython-input-43-e6eb7e39fd31> in <module> ----> 1 data.interp(y=ind)

~/dev/xarray/xarray/core/dataarray.py in interp(self, coords, method, assume_sorted, kwargs, coords_kwargs) 1303 kwargs=kwargs, 1304 assume_sorted=assume_sorted, -> 1305 coords_kwargs 1306 ) 1307 return self._from_temp_dataset(ds)

~/dev/xarray/xarray/core/dataset.py in interp(self, coords, method, assume_sorted, kwargs, coords_kwargs) 2455 } 2456 variables[name] = missing.interp( -> 2457 var, var_indexers, method, kwargs 2458 ) 2459 elif all(d not in indexers for d in var.dims):

~/dev/xarray/xarray/core/missing.py in interp(var, indexes_coords, method, *kwargs) 533 else: 534 out_dims.add(d) --> 535 return result.transpose(tuple(out_dims)) 536 537

~/dev/xarray/xarray/core/variable.py in transpose(self, *dims) 1219 return self.copy(deep=False) 1220 -> 1221 data = as_indexable(self._data).transpose(axes) 1222 return type(self)(dims, data, self._attrs, self._encoding, fastpath=True) 1223

~/dev/xarray/xarray/core/indexing.py in transpose(self, order) 1218 1219 def transpose(self, order): -> 1220 return self.array.transpose(order) 1221 1222 def getitem(self, key):

ValueError: axes don't match array

In [44]: data.reindex(y=ind) /Users/shoyer/dev/xarray/xarray/core/dataarray.py:1240: FutureWarning: Indexer has dimensions ('x', 'z') that are different from that to be indexed along y. This will behave differently in the future. fill_value=fill_value,

ValueError Traceback (most recent call last) <ipython-input-44-1277ead996ae> in <module> ----> 1 data.reindex(y=ind)

~/dev/xarray/xarray/core/dataarray.py in reindex(self, indexers, method, tolerance, copy, fill_value, **indexers_kwargs) 1238 tolerance=tolerance, 1239 copy=copy, -> 1240 fill_value=fill_value, 1241 ) 1242 return self._from_temp_dataset(ds)

~/dev/xarray/xarray/core/dataset.py in reindex(self, indexers, method, tolerance, copy, fill_value, **indexers_kwargs) 2360 tolerance, 2361 copy=copy, -> 2362 fill_value=fill_value, 2363 ) 2364 coord_names = set(self._coord_names)

~/dev/xarray/xarray/core/alignment.py in reindex_variables(variables, sizes, indexes, indexers, method, tolerance, copy, fill_value) 398 ) 399 --> 400 target = new_indexes[dim] = utils.safe_cast_to_index(indexers[dim]) 401 402 if dim in indexes:

~/dev/xarray/xarray/core/utils.py in safe_cast_to_index(array) 104 index = array 105 elif hasattr(array, "to_index"): --> 106 index = array.to_index() 107 else: 108 kwargs = {}

~/dev/xarray/xarray/core/dataarray.py in to_index(self) 545 arrays. 546 """ --> 547 return self.variable.to_index() 548 549 @property

~/dev/xarray/xarray/core/variable.py in to_index(self) 445 def to_index(self): 446 """Convert this variable to a pandas.Index""" --> 447 return self.to_index_variable().to_index() 448 449 def to_dict(self, data=True):

~/dev/xarray/xarray/core/variable.py in to_index_variable(self) 438 """Return this variable as an xarray.IndexVariable""" 439 return IndexVariable( --> 440 self.dims, self._data, self._attrs, encoding=self._encoding, fastpath=True 441 ) 442

~/dev/xarray/xarray/core/variable.py in init(self, dims, data, attrs, encoding, fastpath) 1941 super().init(dims, data, attrs, encoding, fastpath) 1942 if self.ndim != 1: -> 1943 raise ValueError("%s objects must be 1-dimensional" % type(self).name) 1944 1945 # Unlike in Variable, always eagerly load values into memory

ValueError: IndexVariable objects must be 1-dimensional ```

Yes, this is a little silly, but do we have a preferred capitalization for the proper name?

We mostly stick to "xarray" in the docs but "Xarray" or "XArray" is arguably a little more readable and grammatically correct.

Python 2.7 builds on Windows are failing: https://ci.appveyor.com/project/shoyer/xray/build/1.0.3018

The tests that are failing are all variations of test_dask_distributed_integration_test. Example error message: ``` =================================== ERRORS ==================================== _ ERROR at teardown of test_dask_distributed_integration_test[scipy] ____ @pytest.fixture def loop(): with pristine_loop() as loop: # Monkey-patch IOLoop.start to wait for loop stop orig_start = loop.start is_stopped = threading.Event() is_stopped.set() def start(): is_stopped.clear() try: orig_start() finally: is_stopped.set() loop.start = start

        yield loop
        # Stop the loop in case it's still running
        try:
            loop.add_callback(loop.stop)
        except RuntimeError as e:
            if not re.match("IOLoop is clos(ed|ing)", str(e)):
                raise
        else:

          is_stopped.wait()

C:\Python27-conda64\envs\test_env\lib\site-packages\distributed\utils_test.py:102:

C:\Python27-conda64\envs\test_env\lib\contextlib.py:24: in exit self.gen.next() C:\Python27-conda64\envs\test_env\lib\site-packages\distributed\utils_test.py:139: in pristine_loop loop.close(all_fds=True) C:\Python27-conda64\envs\test_env\lib\site-packages\tornado\ioloop.py:716: in close self.remove_handler(self._waker.fileno()) C:\Python27-conda64\envs\test_env\lib\site-packages\tornado\platform\common.py:91: in fileno return self.reader.fileno() C:\Python27-conda64\envs\test_env\lib\socket.py:228: in meth return getattr(self._sock,name)(*args)

args = (<socket._closedsocket object at 0x00000000131F27F0>, 'fileno') def _dummy(*args):

  raise error(EBADF, 'Bad file descriptor')

E error: [Errno 9] Bad file descriptor C:\Python27-conda64\envs\test_env\lib\socket.py:174: error ---------------------------- Captured stderr call ----------------------------- distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:1094 distributed.worker - INFO - Start worker at: tcp://127.0.0.1:1096 distributed.worker - INFO - Start worker at: tcp://127.0.0.1:1095 distributed.worker - INFO - Listening to: tcp://127.0.0.1:1096 distributed.worker - INFO - Listening to: tcp://127.0.0.1:1095 distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:1094 distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:1094 distributed.worker - INFO - ------------------------------------------------- distributed.worker - INFO - ------------------------------------------------- distributed.worker - INFO - Threads: 1 distributed.worker - INFO - Threads: 1 distributed.worker - INFO - Memory: 2.00 GB distributed.worker - INFO - Memory: 2.00 GB distributed.worker - INFO - Local Directory: C:\projects\xray_test_worker-4043f797-3668-459a-9d5b-017dbc092ad5\worker-ozlw8t distributed.worker - INFO - Local Directory: C:\projects\xray_test_worker-0b2d640d-07ba-493f-967c-f8d8de38e3b5\worker-_xbrz6 distributed.worker - INFO - ------------------------------------------------- distributed.worker - INFO - ------------------------------------------------- distributed.scheduler - INFO - Register tcp://127.0.0.1:1096 distributed.worker - INFO - Registered to: tcp://127.0.0.1:1094 distributed.worker - INFO - ------------------------------------------------- distributed.scheduler - INFO - Register tcp://127.0.0.1:1095 distributed.worker - INFO - Registered to: tcp://127.0.0.1:1094 distributed.worker - INFO - ------------------------------------------------- distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:1095 distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:1096 distributed.scheduler - INFO - Receive client connection: Client-06708a40-ce25-11e7-898c-00155d57f2dd distributed.scheduler - INFO - Connection to client Client-06708a40-ce25-11e7-898c-00155d57f2dd broken distributed.scheduler - INFO - Remove client Client-06708a40-ce25-11e7-898c-00155d57f2dd distributed.scheduler - INFO - Close client connection: Client-06708a40-ce25-11e7-898c-00155d57f2dd distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:1095 distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:1096 distributed.scheduler - INFO - Remove worker tcp://127.0.0.1:1095 distributed.scheduler - INFO - Remove worker tcp://127.0.0.1:1096 distributed.scheduler - INFO - Lost all workers distributed.worker - INFO - Close compute stream distributed.worker - INFO - Close compute stream distributed.scheduler - INFO - Scheduler closing... distributed.scheduler - INFO - Scheduler closing all comms ```

@mrocklin any guesses about what this could be?

Fixes #706

New (experimental) decorators xarray.register_dataset_accessor and xarray.register_dataarray_accessor for registering custom xarray extensions without subclassing. They are described in the new documentation page on internals.

CC @rafa-guedes @rabernat @fmaussion @khaeru @ajdawson -- as people who might use such an interface, it would be great to get some feedback about how well this would work for you.

Fixes #367 Fixes #634 Fixes #649

The internal data model used by DataArray has been rewritten to fix several outstanding issues (#367, #634 and this stackoverflow report). Namely, if a DataArray has the same name as one of its coordinates, the array and the coordinate no longer share the same data.

This means that creating a DataArray with the same name as one of its dimensions no longer automatically uses that array to label the corresponding coordinate. You will now need to provide coordinate labels explicitly. Here's the old behavior:

In [2]: xray.DataArray([4, 5, 6], dims='x', name='x') Out[2]: <xray.DataArray 'x' (x: 3)> array([4, 5, 6]) Coordinates: * x (x) int64 4 5 6

and the new behavior (compare the values of the x coordinate):

In [2]: xray.DataArray([4, 5, 6], dims='x', name='x') Out[2]: <xray.DataArray 'x' (x: 3)> array([4, 5, 6]) Coordinates: * x (x) int64 0 1 2

It's also no longer possible to convert a DataArray to a Dataset with DataArray.to_dataset if it is unnamed. This will now raise ValueError. If the array is unnamed, you need to supply the name argument.

Fixes #503

Example usage:

``` In [13]: x = xray.DataArray(np.arange(9).reshape(3, 3), dims=['x', 'y'])

In [14]: x.where(x > 4) Out[14]: <xray.DataArray (x: 3, y: 3)> array([[ nan, nan, nan], [ nan, nan, 5.], [ 6., 7., 8.]]) Coordinates: * y (y) int64 0 1 2 * x (x) int64 0 1 2 ```

Example from "What's new":

``` In [4]: ds = xray.Dataset(coords={'x': range(100), 'y': range(100)})

In [5]: ds['distance'] = np.sqrt(ds.x ** 2 + ds.y ** 2)

In [6]: ds.distance.where(ds.distance < 100).plot() Out[6]: <matplotlib.image.AxesImage at 0x11a819690> ```