This needs proper tests & docs, but would this be a good idea?

A couple of examples are in the docstring. It's mostly just deferring to pandas' excellent eval method.

• [x] Closes #6440 (edit)
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

A couple of tests are failing for the multi-dimensional case, which I'll fix before merge.

What happened?

Very possibly this is user-error, forgive me if so.

I'm trying to transition some code from the previous assignment of MultiIndexes, to the new world. Here's an MCVE:

What did you expect to happen?

No response

Minimal Complete Verifiable Example

```Python da = xr.tutorial.open_dataset("air_temperature")['air']

old code, works, but with a warning

da.expand_dims('foo').assign_coords(foo=(pd.MultiIndex.from_tuples([(1,2)])))

<ipython-input-25-f09b7f52bb42>:1: FutureWarning: the pandas.MultiIndex object(s) passed as 'foo' coordinate(s) or data variable(s) will no longer be implicitly promoted and wrapped into multiple indexed coordinates in the future (i.e., one coordinate for each multi-index level + one dimension coordinate). If you want to keep this behavior, you need to first wrap it explicitly using mindex_coords = xarray.Coordinates.from_pandas_multiindex(mindex_obj, 'dim') and pass it as coordinates, e.g., xarray.Dataset(coords=mindex_coords), dataset.assign_coords(mindex_coords) or dataarray.assign_coords(mindex_coords). da.expand_dims('foo').assign_coords(foo=(pd.MultiIndex.from_tuples([(1,2)]))) Out[25]: <xarray.DataArray 'air' (foo: 1, time: 2920, lat: 25, lon: 53)> array([[[[241.2 , 242.5 , 243.5 , ..., 232.79999, 235.5 , 238.59999], ... [297.69 , 298.09 , 298.09 , ..., 296.49 , 296.19 , 295.69 ]]]], dtype=float32) Coordinates: * lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 22.5 20.0 17.5 15.0 * lon (lon) float32 200.0 202.5 205.0 207.5 ... 325.0 327.5 330.0 * time (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00 * foo (foo) object MultiIndex * foo_level_0 (foo) int64 1 * foo_level_1 (foo) int64 2

new code — seems to get confused between the number of values in the index — 1 — and the number of levels — 3 including the parent:

da.expand_dims('foo').assign_coords(foo=xr.Coordinates.from_pandas_multiindex(pd.MultiIndex.from_tuples([(1,2)]), dim='foo'))

ValueError Traceback (most recent call last) Cell In[26], line 1 ----> 1 da.expand_dims('foo').assign_coords(foo=xr.Coordinates.from_pandas_multiindex(pd.MultiIndex.from_tuples([(1,2)]), dim='foo'))

File ~/workspace/xarray/xarray/core/common.py:621, in DataWithCoords.assign_coords(self, coords, **coords_kwargs) 618 else: 619 results = self._calc_assign_results(coords_combined) --> 621 data.coords.update(results) 622 return data

File ~/workspace/xarray/xarray/core/coordinates.py:566, in Coordinates.update(self, other) 560 # special case for PandasMultiIndex: updating only its dimension coordinate 561 # is still allowed but depreciated. 562 # It is the only case where we need to actually drop coordinates here (multi-index levels) 563 # TODO: remove when removing PandasMultiIndex's dimension coordinate. 564 self._drop_coords(self._names - coords_to_align._names) --> 566 self._update_coords(coords, indexes)

File ~/workspace/xarray/xarray/core/coordinates.py:834, in DataArrayCoordinates._update_coords(self, coords, indexes) 832 coords_plus_data = coords.copy() 833 coords_plus_data[_THIS_ARRAY] = self._data.variable --> 834 dims = calculate_dimensions(coords_plus_data) 835 if not set(dims) <= set(self.dims): 836 raise ValueError( 837 "cannot add coordinates with new dimensions to a DataArray" 838 )

File ~/workspace/xarray/xarray/core/variable.py:3014, in calculate_dimensions(variables) 3012 last_used[dim] = k 3013 elif dims[dim] != size: -> 3014 raise ValueError( 3015 f"conflicting sizes for dimension {dim!r}: " 3016 f"length {size} on {k!r} and length {dims[dim]} on {last_used!r}" 3017 ) 3018 return dims

ValueError: conflicting sizes for dimension 'foo': length 1 on <this-array> and length 3 on {'lat': 'lat', 'lon': 'lon', 'time': 'time', 'foo': 'foo'} ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

This wasn't working on an unregion-ed write; I think because new_var was being lost.

Would it be possible to not have this config? It overrides the local VS Code config, and means VS Code constantly is reporting errors for me.

Totally open to other approaches ofc. Or that we decide that the tradeoff is worthwhile

Also from the chars of #8208 -- this uses the TypeVars we define, which hopefully sets a standard and removes ambiguity for how to type functions. It also allows subclasses.

Where possible, it uses T_Xarray, otherwise T_DataArrayOrSet where it's not possible for mypy to narrow the concrete type down.

What is your issue?

It looks like the doc build is failing on a pandas docstring:

``` /home/docs/checkouts/readthedocs.org/user_builds/xray/conda/8156/lib/python3.10/site-packages/cartopy/io/init.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/50m_physical/ne_50m_coastline.zip warnings.warn(f'Downloading: {url}', DownloadWarning) reading sources... [ 99%] user-guide/reshaping reading sources... [ 99%] user-guide/terminology reading sources... [ 99%] user-guide/time-series reading sources... [ 99%] user-guide/weather-climate reading sources... [100%] whats-new

/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/8156/lib/python3.10/site-packages/pandas/core/indexes/base.py:docstring of pandas.core.indexes.base.Index.join:14: WARNING: Inline literal start-string without end-string. /home/docs/checkouts/readthedocs.org/user_builds/xray/conda/8156/lib/python3.10/site-packages/pandas/core/indexes/base.py:docstring of pandas.core.indexes.base.Index.join:15: WARNING: Inline literal start-string without end-string. looking for now-outdated files... none found ```

(also including the cartopy warning in case that's relevant)

Is this expected? Is anyone familiar enough with the doc build to know whether we can disable warnings from 3rd party modules?

Testing for 3.10 seems fairly low value — is there a realistic case where 3.10 would fail but 3.9 & 3.11 would pass?

Merging this would cut the CI queue...

What is your issue?

I can't work out what is causing this, and can't repro locally, though I've tried to ensure the same things are installed.

The bisect is: - Passes: https://github.com/pydata/xarray/runs/6233389985?check_suite_focus=true - Fails: https://github.com/pydata/xarray/runs/6237267544?check_suite_focus=true

Probably we have to skip it in the meantime, which is a shame

Is there a better way of locking the dependency versions so we can rule that out? I generally don't use conda, and poetry is great at this. Is there a conda equivalent?

Replaces https://github.com/pydata/xarray/pull/6541, which was using a branch in this repo, and so duplicating all the tests — which is fine but makes it difficult to understand what tests are being generated

Is your feature request related to a problem?

Currently https://github.com/pydata/xarray/blob/main/HOW_TO_RELEASE.md has a few steps around the stable branch

Describe the solution you'd like

In our dev call, we discussed the possibility of using main in place of stable and removing the stable branch

IIRC there's something we can do on RTD to make that replacement. (If anyone knows to hand, comment here; otherwise I can search for it).

Is there anything else we need to do apart from RTD?

Describe alternatives you've considered

No response

Additional context

No response

A small typing change

• [x] Seems to solve issues in https://github.com/pydata/xarray/pull/6148

• [x] Passes pre-commit run --all-files

The new version of black formats notebook files too, so I was getting diffs when doing other work. This should let us upgrade black in pre-commit when that comes out

• [x] Closes https://github.com/pydata/xarray/issues/5755
• [x] Passes pre-commit run --all-files
• [ ] New functions/methods are listed in api.rst

ref https://github.com/pydata/xarray/issues/5654

I vote to always do this when a dependency changes, if we can't immediately fix the code. Having main be green means it's easy to see the difference between passing and failing.

• [x] Closes https://github.com/pydata/xarray/issues/5675
• [x] Passes pre-commit run --all-files
• [ ] New functions/methods are listed in api.rst

& removed internal usages

• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

I needed to pass indexes=None from some of the existing methods that were using the deprecated method. I spent some time debugging but couldn't work out the causality. I added TODOs as part of the index work.

This consistently fails.

In general I think as soon as we see this happening, we should either fix immediately (but probably not that easy) or xfail it and if necessary open an issue. That keeps the main branch green, such that we can reliably use passing tests as an indicator of PRs.

Though as ever, very open to other thoughts. There's a reasonable argument that "keeping master green by removing any failing tests is missing the point a bit"...

• [x] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

~All apart from this issue, which is a puzzle. It would be great to remove the function, which requires solving that puzzle!~

Hacked around it, somewhat unsatisfactorily. But no need to delay this issue because of it.

• [x] Closes https://github.com/pydata/xarray/issues/5173
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] Add docstring
• [x] Confirm apply_ufunc args, like join

I think this is the right approach — we may need to confirm the appropriate args to apply_ufunc — probably we want the same as .where?

Tests are OK but a bit lacking atm.

• [ ] Closes #xxxx
• [ ] Passes black . && mypy . && flake8
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

I did this before I realized it only works on mypy master! But posting so no one else does it. We can wait to merge until the next version of mypy is out

Let me know any changes here, fairly soon if possible!

And please make changes liberally — what are the important features we should highlight in the heading?

Thanks

As per https://github.com/pydata/xarray/pull/4440

• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

I found the existing example not that clear as to why it was selecting [[1,1],[5,5]], I think this is more obvious (open to disagreement of course!)

Before ```python In [37]: da = xr.DataArray( ....: np.arange(12).reshape((3, 4)), ....: dims=["x", "y"], ....: coords={"x": [0, 1, 2], "y": ["a", "b", "c", "d"]}, ....: ) ....:

In [38]: da Out[38]: <xarray.DataArray (x: 3, y: 4)> array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) Coordinates: * x (x) int64 0 1 2 * y (y) <U1 'a' 'b' 'c' 'd'

In [39]: da[[0, 1], [1, 1]] Out[39]: <xarray.DataArray (x: 2, y: 2)> array([[1, 1], [5, 5]]) Coordinates: * x (x) int64 0 1 * y (y) <U1 'b' 'b' ```

After: ```python

In [4]: da[[1, 2, 2], [0, 3]] Out[4]: <xarray.DataArray (x: 3, y: 2)> array([[ 4, 7], [ 8, 11], [ 8, 11]]) Coordinates: * x (x) int64 1 2 2 * y (y) <U1 'a' 'd'` ```

• [x] Fixes #3760
• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Closes doesn't seem to add the issue to "Linked Issues" and auto-close. Fixes works elsewhere.

• [x] Triages https://github.com/pydata/xarray/issues/3711

Branching from https://github.com/pydata/xarray/issues/3770

Here's the proposal: allow lambdas on methods where the primary argument is a single xarray object, and interpret lambas as though they'd be supplied in a pipe method followed by the current method. Taking the example from the linked issue:

```python

In [1]: import xarray as xr

In [2]: import numpy as np

In [3]: da = xr.DataArray(np.random.rand(2,3))

In [4]: da.where(da > 0.5)
Out[4]: <xarray.DataArray (dim_0: 2, dim_1: 3)> array([[ nan, 0.71442406, nan], [0.55748705, nan, nan]]) Dimensions without coordinates: dim_0, dim_1

this should be equivalent (currently not valid)

In [5]: da.where(lambda x: x > 0.5)

the longer version (currently works)

In [5]: da.pipe(lambda x: x.where(x > 0.5))

```

Others I miss from pandas: assign, and loc. I haven't gone through the list though assume there are others; we don't have to agree 100% on the list before starting with the most obvious ones, assuming we're in agreement with the principle.

• [x] Closes https://github.com/pydata/xarray/issues/3833
• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

I struggled to parse some of the code in the docs, and so formatted some of the hardest-to-read code (e.g. mask = (ds.coords['lat']>20)&(ds.coords['lat']<60)&(ds.coords['lon']>220)&(ds.coords['lon']<260)!) with black.

I'm not sure whether there's a more automated solution for this?

On a related point, I'm not sure why some of the code examples are colorized and some are B&W? I couldn't see anything obvious in the rst.

Any ideas why __hdf5libversion__ wouldn't be available? Shall I put a try / except around it?

```python In [4]: import xarray as xr

In [5]: xr.show_versions()

ModuleNotFoundError Traceback (most recent call last) /usr/local/lib/python3.7/site-packages/xarray/util/print_versions.py in netcdf_and_hdf5_versions() 64 try: ---> 65 import netCDF4 66 libhdf5_version = netCDF4.hdf5libversion

ModuleNotFoundError: No module named 'netCDF4'

During handling of the above exception, another exception occurred:

AttributeError Traceback (most recent call last) <ipython-input-5-6f391305f2fe> in <module> ----> 1 xr.show_versions()

/usr/local/lib/python3.7/site-packages/xarray/util/print_versions.py in show_versions(file) 78 sys_info = get_sys_info() 79 ---> 80 sys_info.extend(netcdf_and_hdf5_versions()) 81 82 deps = [

/usr/local/lib/python3.7/site-packages/xarray/util/print_versions.py in netcdf_and_hdf5_versions() 69 try: 70 import h5py ---> 71 libhdf5_version = h5py.hdf5libversion 72 except ImportError: 73 pass

AttributeError: module 'h5py' has no attribute 'hdf5libversion' ```

I check I'm on the latest h5py: pip install h5py -U Thu Jul 18 16:47:29 2019 Requirement already up-to-date: h5py in ./Library/Python/3.7/lib/python/site-packages (2.9.0) Requirement already satisfied, skipping upgrade: numpy>=1.7 in /usr/local/lib/python3.7/site-packages (from h5py) (1.16.4) Requirement already satisfied, skipping upgrade: six in ./Library/Python/3.7/lib/python/site-packages (from h5py) (1.12.0)

• [x] Tests added
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

I need to confirm whether this works with an installed version of xarray, because of https://github.com/pytest-dev/pytest/issues/1596. If pytest doesn't pick up this config, it will run all tests, because the default is to not skip.

I know I've generally been the point person for pytest - let me know if anyone has an immediate solution for this though

Should I still add a whatsnew? It's very small and very internal

is relies on interning, which is almost always OK for small strings, but reliant on language implementations

(unless there's something I'm missing about .kind?)

This is extremely minor, I generally wouldn't report it.

We're using classes inherited from Dataset more & more - this works really well for classes with a lot of array-like properties that can be aligned, and allows @property to lazily compute some calcs. (any feedback on this approach very welcome)

The top of the repr is incorrect

python <xarray.SecurityMeasure> Dimensions: (date: 6647, security: 285, sub_measure: 4) Coordinates: ...

Could just be the qualified name.

• [x] Closes https://github.com/pydata/xarray/issues/2627

```python

In [1]: import xarray as xr

In [3]: ds=xr.Dataset({'a':xr.DataArray([1,2,3])})

In [4]: xr.Dataset(ds) /Users/maximilian/drive/workspace/xarray/xarray/core/dataset.py:373: FutureWarning: iteration over an xarray.Dataset will change in xarray v0.11 to only include data variables, not coordinates. Iterate over the Dataset.variables property instead to preserve existing behavior in a forwards compatible manner. both_data_and_coords = [k for k in data_vars if k in coords] Out[4]: <xarray.Dataset> Dimensions: (dim_0: 3) Dimensions without coordinates: dim_0 Data variables: a (dim_0) int64 1 2 3 ```

Problem description

Currently we run both_data_and_coords = [k for k in data_vars if k in coords], where data_vars is the first arg to the constructor. So, when data_vars is a Dataset, we'll raise a warning.

More importantly: this raises the question of how we should handle xr.Dataset(ds) Currently we retain the coords, but discard attrs. That seems arbitrary? I'd have to check, but when we change __iter__ to only reference data_vars, we may also discard coords

I don't have a strong view. If you think that Foo(foo) should always return foo (which is often the case in pandas), then we should retain the full object. If you think that foo is supplied to the data_vars arg, then retaining only the data_vars seems reasonable

Expected Output

Output of xr.show_versions()

I originally posted this on SO, as I thought it was a user question rather than a library issue. But after working on it more today, I'm not so sure.

I'm trying to do a 'join' in xarray, but using a non-index coordinate rather than a shared dim.

I have a Dataset indexed on 'a' with a coord on 'b', and a DataArray indexed on 'b':

```python In [17]: ds=xr.Dataset(dict(a=(('x'),np.random.rand(10))), coords=dict(b=(('x'),list(range(10)))))

In [18]: ds Out[18]: <xarray.Dataset> Dimensions: (x: 10) Coordinates: b (x) int64 0 1 2 3 4 5 6 7 8 9 Dimensions without coordinates: x Data variables: a (x) float64 0.3634 0.2132 0.6945 0.5359 0.1053 0.07045 0.5945 ...

In [19]: da=xr.DataArray(np.random.rand(10), dims=('b',), coords=dict(b=(('b'),list(range(10)))))

In [20]: da Out[20]: <xarray.DataArray (b: 10)> array([0.796987, 0.275992, 0.747882, 0.240374, 0.435143, 0.285271, 0.753582, 0.556038, 0.365889, 0.434844]) Coordinates: * b (b) int64 0 1 2 3 4 5 6 7 8 9

```

Can I add da onto my dataset, by joining on ds.b equalling da.b? The result would be:

python <xarray.Dataset> Dimensions: (x: 10) Coordinates: b (x) int64 0 1 2 3 4 5 6 7 8 9 Dimensions without coordinates: x Data variables: a (x) float64 0.3634 0.2132 0.6945 0.5359 0.1053 0.07045 0.5945 ... da (x) float64 0.796987, 0.275992, 0.747882, 0.240374, 0.435143 ...

(for completeness - the data isn't current in the correct position)

• [x] Closes #884 (remove if there is no corresponding issue, which should only be the case for minor changes)
• [x] Tests added (for all bug fixes or enhancements)
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

I tried setting this up on GCS as an experiment, and found there's no mode kwarg to open_zarr. I'm not sure whether there are broader issues but thought I'd put this in regardless

(also - if anyone has any experience in deploying xarray over https://github.com/dask/dask-kubernetes and GCS, I'd be interested in your experience. We're currently working with either a) BigQuery for big data or b) in-memory xarray for small data, and I was interested whether there's a way of retaining the xarray model for bigger data without too much overhead)

http://xray.readthedocs.org/en/stable/generated/xray.DataArray.count.html

name doesn't seem to be working correctly:

Dimension(s) over which to apply {name}.

This is the repr for a DataArray with 7 coordinates and 5 dimensions. Do we agree this should be shorter?

``` python <xray.DataArray (a: 2, b: 2, c: 5, d: 6771, e: 58)> array([[[[[ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], ..., [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]]],


    [[[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]]]],



   [[[[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]]],


    [[[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]]]]])

Coordinates: ... ```

Even at the beginning of this year its downloads were 90% lower than 2.6: https://hynek.me/articles/python3-2016/

Impetus is that pytest no longer supports it, and some of the tests I recently wrote don't work on 3.3

For an item to be a value in a Dataset, it either needs to be array-like, or pass lib.is_scalar.

I think this is probably too strict - I'd propose anything can be a value, and it only gets treated as an array if it looks like one - thoughts?

Here's the code that checks the values: https://github.com/pydata/xarray/blob/master/xarray/core/variable.py#L52

``` python In [13]: class B(object): pass ....:

In [14]: xr.Dataset({'a':B()})

ValueError Traceback (most recent call last) <ipython-input-14-b2eeb6de19ea> in <module>() ----> 1 xr.Dataset({'a':B()})

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/xarray/core/dataset.py in init(self, data_vars, coords, attrs, compat, **kwargs) 207 coords = set() 208 if data_vars is not None or coords is not None: --> 209 self._set_init_vars_and_dims(data_vars, coords, compat) 210 if attrs is not None: 211 self.attrs = attrs

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/xarray/core/dataset.py in _set_init_vars_and_dims(self, vars, coords, compat) 258 aligned = align_variables(variables) 259 new_variables, new_coord_names = expand_variables(aligned, --> 260 compat=compat) 261 262 new_coord_names.update(coords)

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/xarray/core/merge.py in expand_variables(raw_variables, old_variables, compat) 75 add_variable(dim, coord.variable) 76 var = var.variable ---> 77 add_variable(name, var) 78 79 return new_variables, new_coord_names

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/xarray/core/merge.py in add_variable(name, var) 55 56 def add_variable(name, var): ---> 57 var = _as_dataset_variable(name, var) 58 if name not in variables: 59 variables[name] = var

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/xarray/core/merge.py in _as_dataset_variable(name, var) 9 """ 10 try: ---> 11 var = as_variable(var, key=name) 12 except TypeError: 13 raise TypeError('variables must be given by arrays or a tuple of '

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/xarray/core/variable.py in as_variable(obj, key, strict, copy) 55 obj = Variable(obj.name, obj) 56 elif key is not None: ---> 57 obj = Variable(key, obj) 58 else: 59 raise TypeError('cannot infer Variable dimensions')

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/xarray/core/variable.py in init(self, dims, data, attrs, encoding, fastpath) 211 """ 212 self._data = as_compatible_data(data, fastpath=fastpath) --> 213 self._dims = self._parse_dimensions(dims) 214 self._attrs = None 215 self._encoding = None

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/xarray/core/variable.py in _parse_dimensions(self, dims) 319 raise ValueError('dimensions %s must have the same length as the ' 320 'number of data dimensions, ndim=%s' --> 321 % (dims, self.ndim)) 322 return dims 323

ValueError: dimensions ('a',) must have the same length as the number of data dimensions, ndim=0 ```

Inspired by: https://github.com/pydata/xarray/pull/844. This is a more general case for variable selection.

We could allow a selection of variables with a callable, similar (but not the same) as pandas' implementation:

``` python In [5]: ds=xr.Dataset({'a': (('x', 'y'), np.random.rand(10,2))}) Out[4]: <xarray.Dataset> Dimensions: (x: 10, y: 2) Coordinates: * x (x) int64 0 1 2 3 4 5 6 7 8 9 * y (y) int64 0 1 Data variables: a (x, y) float64 0.5819 0.1214 0.2645 0.9053 0.6968 0.1608 0.3199 ...

In [9]: ds['a'].attrs['clean'] = True

potentially:

In [10]: ds[lambda x: x.attrs['clean']]

... would return ds['a']

```

This would mean functions wouldn't be able to be dataset keys - I don't think that's a big issue, but could arise with callable classes, for example.

Another option would be a .filter / .select method or indexer.

What do we think about switching to py.test?

Current tests pass with py.test. Then we can use its features for new tests, while the old ones will continue working.

Resolves https://github.com/pydata/xarray/issues/745

A bit of a 'convenient hack'. Let me know if you think there's a better way to do this

Closes https://github.com/xray/xray/issues/697

@shoyer: am I missing something as to why the special case was there?

closes https://github.com/xray/xray/issues/664. What's new & docs forthcoming

We have a pandas DataFrame which is not aligned on an xray Dataset:

``` python In [34]:

da = xray.DataArray( np.random.rand(5,2), coords=( ('date', pd.date_range(start='2000', periods=5)), ('company', list('ab')), ) ) da Out[34]: <xray.DataArray (date: 5, company: 2)> array([[ 0.82168647, 0.93097023], [ 0.34928855, 0.23245631], [ 0.32857461, 0.12554705], [ 0.44983381, 0.27182767], [ 0.31063147, 0.52894834]]) Coordinates: * date (date) datetime64[ns] 2000-01-01 2000-01-02 2000-01-03 ... * company (company) |S1 'a' 'b' In [35]:

ds = xray.Dataset({'returns': da}) ds Out[35]: <xray.Dataset> Dimensions: (company: 2, date: 5) Coordinates: * date (date) datetime64[ns] 2000-01-01 2000-01-02 2000-01-03 ... * company (company) |S1 'a' 'b' Data variables: returns (date, company) float64 0.8217 0.931 0.3493 0.2325 0.3286 ... In [36]:

df=da.to_pandas() df Out[36]: company a b date
2000-01-01 0.821686 0.930970 2000-01-02 0.349289 0.232456 2000-01-03 0.328575 0.125547 2000-01-04 0.449834 0.271828 2000-01-05 0.310631 0.528948 In [41]:

rank rank = df.rank() rank Out[41]: company a b date
2000-01-01 5 5 2000-01-02 3 2 2000-01-03 2 1 2000-01-04 4 3 2000-01-05 1 4 In [42]:

rank=rank.reindex(columns=list('ba')) rank Out[42]: company b a date
2000-01-01 5 5 2000-01-02 2 3 2000-01-03 1 2 2000-01-04 3 4 2000-01-05 4 1 ```

When we add it to a Dataset, it ignores the index on the columns:

``` python In [49]:

ds['rank'] = (('date','company'),rank) ds['rank'].to_pandas() Out[49]: company a b date
2000-01-01 5 5 2000-01-02 2 3 2000-01-03 1 2 2000-01-04 3 4 2000-01-05 4 1 ```

And adding the DataFrame without supplying dims doesn't work. One solution, is to construct a DataArray out of the pandas object:

``` python In [45]:

ds['rank'] = xray.DataArray(rank) ds Out[45]: <xray.Dataset> Dimensions: (company: 2, date: 5) Coordinates: * date (date) datetime64[ns] 2000-01-01 2000-01-02 2000-01-03 ... * company (company) object 'a' 'b' Data variables: returns (date, company) float64 0.8217 0.931 0.3493 0.2325 0.3286 ... rank (date, company) float64 5.0 5.0 3.0 2.0 2.0 1.0 4.0 3.0 1.0 4.0 ```

Possible additions to make this easier: - Align pandas objects that are passed in, when dims are supplied - Allow adding pandas objects to Datasets with labelled axes without supplying dims, and align those (similar to wrapping them in a DataArray constructor)

What are your thoughts?