What is your issue?

I've recently been hit with this when experimenting with xr.dot and xr.corr — xr.dot takes dims, and xr.cov takes dim. Because they each take multiple arrays as positional args, kwargs are more conventional.

Should we standardize on one of these?

What happened?

Writing with region with chunks that aren't aligned can lose data.

I've recreated an example below. While it's unlikely that folks are passing different values to .chunk for the template vs. the regions, I had an "auto" chunk, which can then set different chunk values.

(FWIW, this was fairly painful, and I managed to lose a lot of time by not noticing this, and then not really considering this could happen as I was trying to debug. I think we should really strive to ensure that we don't lose data / incorrectly report that we've successfully written data...)

What did you expect to happen?

If there's a risk of data loss, raise an error...

Minimal Complete Verifiable Example

```Python ds = xr.DataArray(np.arange(120).reshape(4,3,-1),dims=list("abc")).rename('var1').to_dataset().chunk(2)

ds

<xarray.Dataset>

Dimensions: (a: 4, b: 3, c: 10)

Dimensions without coordinates: a, b, c

Data variables:

var1 (a, b, c) int64 dask.array<chunksize=(2, 2, 2), meta=np.ndarray>

def write(ds): ds.chunk(5).to_zarr('foo.zarr', compute=False, mode='w') for r in (range(ds.sizes['a'])): ds.chunk(3).isel(a=[r]).to_zarr('foo.zarr', region=dict(a=slice(r, r+1)))

def read(ds): result = xr.open_zarr('foo.zarr') assert result.compute().identical(ds) print(result.chunksizes, ds.chunksizes)

write(ds); read(ds)

AssertionError

xr.open_zarr('foo.zarr').compute()['var1']

<xarray.DataArray 'var1' (a: 4, b: 3, c: 10)> array([[[ 0, 0, 0, 3, 4, 5, 0, 0, 0, 9], [ 0, 0, 0, 13, 14, 15, 0, 0, 0, 19], [ 0, 0, 0, 23, 24, 25, 0, 0, 0, 29]],

   [[ 30,  31,  32,   0,   0,  35,  36,  37,  38,   0],
    [ 40,  41,  42,   0,   0,  45,  46,  47,  48,   0],
    [ 50,  51,  52,   0,   0,  55,  56,  57,  58,   0]],

   [[ 60,  61,  62,   0,   0,  65,   0,   0,   0,  69],
    [ 70,  71,  72,   0,   0,  75,   0,   0,   0,  79],
    [ 80,  81,  82,   0,   0,  85,   0,   0,   0,  89]],

   [[  0,   0,   0,  93,  94,  95,  96,  97,  98,   0],
    [  0,   0,   0, 103, 104, 105, 106, 107, 108,   0],
    [  0,   0,   0, 113, 114, 115, 116, 117, 118,   0]]])

Dimensions without coordinates: a, b, c ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Is your feature request related to a problem?

Would having the nbytes value in the Dataset repr be reasonable?

I frequently find myself logging this separately. For example:

diff <xarray.Dataset> Dimensions: (lat: 25, time: 2920, lon: 53) Coordinates: * lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0 * lon (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0 * time (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00 Data variables: - air (time, lat, lon) float32 dask.array<chunksize=(2920, 25, 53), meta=np.ndarray> + air (time, lat, lon) float32 15MB dask.array<chunksize=(2920, 25, 53), meta=np.ndarray> Attributes: Conventions: COARDS title: 4x daily NMC reanalysis (1948) description: Data is from NMC initialized reanalysis\n(4x/day). These a... platform: Model references: http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanaly...

Describe the solution you'd like

No response

Describe alternatives you've considered

Status quo :)

Additional context

No response

What is your issue?

Sorry I haven't been on the calls at all recently (unfortunately the schedule is difficult for me). Maybe this was discussed there? 

PRs now seem to require a separate approval prior to merging. Is there an upside to this? Is there any difference between those who can approve and those who can merge? Otherwise it just seems like more clicking.

PRs also now seem to require merging the latest main prior to merging? I get there's some theoretical value to this, because changes can semantically conflict with each other. But it's extremely rare that this actually happens (can we point to cases?), and it limits the immediacy & throughput of PRs. If the bad outcome does ever happen, we find out quickly when main tests fail and can revert.

(fwiw I wrote a few principles around this down a while ago here; those are much stronger than what I'm suggesting in this issue though)

What happened?

When writing a coord with a datetime type, the chunking on the coord is ignored, and the whole coord is written as a single chunk. (or at least it can be, I haven't done enough to confirm whether it'll always be...)

This can be quite inconvenient. Any attempt to write to that dataset from a distributed process will have errors, since each process will be attempting to write another process's data, rather than only its region. And less severely, the chunks won't be unified.

Minimal Complete Verifiable Example

```Python ds = xr.tutorial.load_dataset('air_temperature')

( ds.chunk() .expand_dims(a=1000) .assign_coords( time2=lambda x: x.time, time_int=lambda x: (("time"), np.full(ds.sizes["time"], 1)), ) .chunk(time=10) .to_zarr("foo.zarr", mode="w") )

xr.open_zarr('foo.zarr')

Note the chunksize=(2920,) vs chunksize=(10,)!

<xarray.Dataset> Dimensions: (a: 1000, time: 2920, lat: 25, lon: 53) Coordinates: * lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 22.5 20.0 17.5 15.0 * lon (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0 * time (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00 time2 (time) datetime64[ns] dask.array<chunksize=(2920,), meta=np.ndarray> # here time_int (time) int64 dask.array<chunksize=(10,), meta=np.ndarray> # here Dimensions without coordinates: a Data variables: air (a, time, lat, lon) float32 dask.array<chunksize=(1000, 10, 25, 53), meta=np.ndarray> Attributes: Conventions: COARDS description: Data is from NMC initialized reanalysis\n(4x/day). These a... platform: Model references: http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanaly... title: 4x daily NMC reanalysis (1948)

xr.open_zarr('foo.zarr').chunks

ValueError Traceback (most recent call last) Cell In[13], line 1 ----> 1 xr.open_zarr('foo.zarr').chunks

File /opt/homebrew/lib/python3.9/site-packages/xarray/core/dataset.py:2567, in Dataset.chunks(self) 2552 @property 2553 def chunks(self) -> Mapping[Hashable, tuple[int, ...]]: 2554 """ 2555 Mapping from dimension names to block lengths for this dataset's data, or None if 2556 the underlying data is not a dask array. (...) 2565 xarray.unify_chunks 2566 """ -> 2567 return get_chunksizes(self.variables.values())

File /opt/homebrew/lib/python3.9/site-packages/xarray/core/common.py:2013, in get_chunksizes(variables) 2011 for dim, c in v.chunksizes.items(): 2012 if dim in chunks and c != chunks[dim]: -> 2013 raise ValueError( 2014 f"Object has inconsistent chunks along dimension {dim}. " 2015 "This can be fixed by calling unify_chunks()." 2016 ) 2017 chunks[dim] = c 2018 return Frozen(chunks)

ValueError: Object has inconsistent chunks along dimension time. This can be fixed by calling unify_chunks().

```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

What is your issue?

We raise an error on grouping without supplying dims, but not for float indexes — is this intentional or an oversight?

This is without flox installed

```python

da = xr.tutorial.open_dataset("air_temperature")['air']

da.drop_vars('lat').groupby('lat').sum() ```

```

ValueError Traceback (most recent call last) Cell In[8], line 1 ----> 1 da.drop_vars('lat').groupby('lat').sum() ... ValueError: cannot reduce over dimensions ['lat']. expected either '...' to reduce over all dimensions or one or more of ('time', 'lon'). ```

But with a float index, we don't raise:

python da.groupby('lat').sum()

...returns the original array:

Out[15]: <xarray.DataArray 'air' (time: 2920, lat: 25, lon: 53)> array([[[296.29 , 296.79 , 297.1 , ..., 296.9 , 296.79 , 296.6 ], [295.9 , 296.19998, 296.79 , ..., 295.9 , 295.9 , 295.19998], [296.6 , 296.19998, 296.4 , ..., 295.4 , 295.1 , 294.69998], ...

And if we try this with a non-float index, we get the error again:

python da.groupby('time').sum()

ValueError: cannot reduce over dimensions ['time']. expected either '...' to reduce over all dimensions or one or more of ('lat', 'lon').

What happened?

I'm getting task graphs > 1GB, I think possibly because the full indexes are being included in every task?

What did you expect to happen?

Only the relevant sections of the index would be included

Minimal Complete Verifiable Example

```Python da = xr.tutorial.load_dataset('air_temperature')

Dropping the index doesn't generally matter that much...

len(cloudpickle.dumps(da.chunk(lat=1, lon=1)))

15569320

len(cloudpickle.dumps(da.chunk().drop_vars(da.indexes)))

15477313

But with .map_blocks, it really matters — it's really big with the indexes, and the same size without:

len(cloudpickle.dumps(da.chunk(lat=1, lon=1).map_blocks(lambda x: x)))

79307120

len(cloudpickle.dumps(da.chunk(lat=1, lon=1).drop_vars(da.indexes).map_blocks(lambda x: x)))

16016173

```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Is your feature request related to a problem?

Currently I have a workload which does something a bit like:

python ds = open_zarr(source) ( ds.assign( x=ds.foo * ds.bar y=ds.foo + ds.bar ).to_zarr(dest) )

(the actual calc is a bit more complicated! And while I don't have a MVCE of the full calc, I pasted a task graph below)

Dask — while very impressive in many ways — handles this extremely badly, because it attempts to load the whole of ds into memory before writing out any chunks. There are lots of issues on this in the dask repo; it seems like an intractable problem for dask.

Describe the solution you'd like

I was hoping to make the internals of this task opaque to dask, so it became a much dumber task runner — just map over the blocks, running the function and writing the result, block by block. I thought I had some success with .map_blocks last week — the internals of the calc are now opaque at least. But the dask cluster is falling over again, I think because the write is seen as a separate task.

Is there any way to make the write more opaque too?

Describe alternatives you've considered

I've built a homegrown thing which is really hacky which does this on a custom scheduler — just runs the functions and writes with region. I'd much prefer to use & contribute to the broader ecosystem...

Additional context

(It's also possible I'm making some basic error — and I do remember it working much better last week — so please feel free to direct me / ask me for more examples, if this doesn't ring true)

Is your feature request related to a problem? Please describe.

Pandas has a .expanding aggregation, which is basically rolling with a full lookback. I often end up supplying rolling with the length of the dimension, and this is some nice sugar for that.

Describe the solution you'd like Basically the same as pandas — a .expanding method that returns an Expanding class, which implements the same methods as a Rolling class.

Describe alternatives you've considered Some options: – This – Don't add anything, the sugar isn't worth the additional API. – Go full out and write specialized expanding algos — which will be faster since they don't have to keep track of the window. But not that much faster, likely not worth the effort.

What is your issue?

Currently a DataArray's dims are ordered, while a Dataset's are not.

Do we gain anything from have unordered dims in a Dataset? Could we have an ordering without enforcing it on every variable?

Here's one proposal, with fairly wide error-bars: - Datasets have a dim order, which is set at construction time or through .transpose - Currently .transpose changes the order of each variable's dims, but not the dataset's - If dims aren't supplied, we can just use the first variable's - Variables don't have to conform to that order — .assign(foo=differently_ordered) maintains the differently ordered dims. So this doesn't limit any current functionality. - When there are transformations which change dim ordering, Xarray is "allowed" to transpose variables to the dataset's ordering. Currently Xarray is "allowed" to change dim order arbitrarily — for example to put a core dim last. IIUC, we'd prefer to set a non-arbitrary order, but we don't have one to reference. - This would remove a bunch of boilerplate from methods that save the ordering, run .apply_ufunc and then reorder in the original order[^1]

What do folks think?

[^1]: though also we could do this in .apply_ufunc

What is your issue?

Any ideas what could be causing these? I can't immediately reproduce locally.

https://github.com/pydata/xarray/actions/runs/7105414268/job/19342564583

``` Error: TestDataArray.test_computation_objects[int64-method_groupby_bins-data]

AssertionError: Left and right DataArray objects are not close

Differing values: L <Quantity([[ nan nan 1. 1. ] [2. 2. 3. 3. ] [4. 4. 5. 5. ] [6. 6. 7. 7. ] [8. 8. 9. 9.333333]], 'meter')> R <Quantity([[0. 0. 1. 1. ] [2. 2. 3. 3. ] [4. 4. 5. 5. ] [6. 6. 7. 7. ] [8. 8. 9. 9.333333]], 'meter')> ```

Is your feature request related to a problem?

We currently have query, which can runs a numexpr string using eval.

Describe the solution you'd like

Should we add an eval method itself? I find that when building something for the command line, allowing people to pass an eval-able expression can be a good interface.

Describe alternatives you've considered

No response

Additional context

No response

I was attempting to copy the great work here for numbagg and spotted this! Do we fire twice for each release? Maybe that's fine though?

https://github.com/pydata/xarray/actions/workflows/pypi-release.yaml

Is your feature request related to a problem? Please describe.

A method like .reset_index has a signature .reset_index(dims_or_levels, drop=False).

This means that passing .reset_index("x", "y") is actually like passing .reset_index("x", True), which is silent and confusing.

Describe the solution you'd like Move to kwarg-only arguments for these; like .reset_index(dims_or_levels, *, drop=False).

But we probably need a deprecation cycle, which will require some work.

Describe alternatives you've considered Not have a deprecation cycle? I imagine it's fairly rare to not pass the kwarg.

Is your feature request related to a problem?

I'd like an exponentially moving correlation coefficient

Describe the solution you'd like

I think we could add a rolling_exp.corr method fairly easily — i.e. just in python, no need to add anything to numbagg: ewma here means rolling_exp(...).mean - ewma(A * B) - ewma(A) * ewma(B) for the rolling covar - divided by sqrt of (ewma(A**2) - ewma(A)**2 * ewma(B**2) - ewma(B)**2 for the sqrt of variance

We could also add a flag for cosine similarity, which wouldn't remove the mean. We could also add .var & .std & .covar as their own methods.

I think we'd need to mask the variables on their intersection, so we don't have values that are missing from B affecting A's variance without affecting its covariance.

Pandas does this in cython, possibly because it's faster to only do a single pass of the data. If anyone has correctness concerns about this simple approach of wrapping ewmas, please let me know. Or if the performance would be unacceptable such that it shouldn't go into xarray until it's a single pass.

Describe alternatives you've considered

Numagg

Additional context

No response

What happened?

Very possibly this is user-error, forgive me if so.

I'm trying to transition some code from the previous assignment of MultiIndexes, to the new world. Here's an MCVE:

What did you expect to happen?

No response

Minimal Complete Verifiable Example

```Python da = xr.tutorial.open_dataset("air_temperature")['air']

old code, works, but with a warning

da.expand_dims('foo').assign_coords(foo=(pd.MultiIndex.from_tuples([(1,2)])))

<ipython-input-25-f09b7f52bb42>:1: FutureWarning: the pandas.MultiIndex object(s) passed as 'foo' coordinate(s) or data variable(s) will no longer be implicitly promoted and wrapped into multiple indexed coordinates in the future (i.e., one coordinate for each multi-index level + one dimension coordinate). If you want to keep this behavior, you need to first wrap it explicitly using mindex_coords = xarray.Coordinates.from_pandas_multiindex(mindex_obj, 'dim') and pass it as coordinates, e.g., xarray.Dataset(coords=mindex_coords), dataset.assign_coords(mindex_coords) or dataarray.assign_coords(mindex_coords). da.expand_dims('foo').assign_coords(foo=(pd.MultiIndex.from_tuples([(1,2)]))) Out[25]: <xarray.DataArray 'air' (foo: 1, time: 2920, lat: 25, lon: 53)> array([[[[241.2 , 242.5 , 243.5 , ..., 232.79999, 235.5 , 238.59999], ... [297.69 , 298.09 , 298.09 , ..., 296.49 , 296.19 , 295.69 ]]]], dtype=float32) Coordinates: * lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 22.5 20.0 17.5 15.0 * lon (lon) float32 200.0 202.5 205.0 207.5 ... 325.0 327.5 330.0 * time (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00 * foo (foo) object MultiIndex * foo_level_0 (foo) int64 1 * foo_level_1 (foo) int64 2

new code — seems to get confused between the number of values in the index — 1 — and the number of levels — 3 including the parent:

da.expand_dims('foo').assign_coords(foo=xr.Coordinates.from_pandas_multiindex(pd.MultiIndex.from_tuples([(1,2)]), dim='foo'))

ValueError Traceback (most recent call last) Cell In[26], line 1 ----> 1 da.expand_dims('foo').assign_coords(foo=xr.Coordinates.from_pandas_multiindex(pd.MultiIndex.from_tuples([(1,2)]), dim='foo'))

File ~/workspace/xarray/xarray/core/common.py:621, in DataWithCoords.assign_coords(self, coords, **coords_kwargs) 618 else: 619 results = self._calc_assign_results(coords_combined) --> 621 data.coords.update(results) 622 return data

File ~/workspace/xarray/xarray/core/coordinates.py:566, in Coordinates.update(self, other) 560 # special case for PandasMultiIndex: updating only its dimension coordinate 561 # is still allowed but depreciated. 562 # It is the only case where we need to actually drop coordinates here (multi-index levels) 563 # TODO: remove when removing PandasMultiIndex's dimension coordinate. 564 self._drop_coords(self._names - coords_to_align._names) --> 566 self._update_coords(coords, indexes)

File ~/workspace/xarray/xarray/core/coordinates.py:834, in DataArrayCoordinates._update_coords(self, coords, indexes) 832 coords_plus_data = coords.copy() 833 coords_plus_data[_THIS_ARRAY] = self._data.variable --> 834 dims = calculate_dimensions(coords_plus_data) 835 if not set(dims) <= set(self.dims): 836 raise ValueError( 837 "cannot add coordinates with new dimensions to a DataArray" 838 )

File ~/workspace/xarray/xarray/core/variable.py:3014, in calculate_dimensions(variables) 3012 last_used[dim] = k 3013 elif dims[dim] != size: -> 3014 raise ValueError( 3015 f"conflicting sizes for dimension {dim!r}: " 3016 f"length {size} on {k!r} and length {dims[dim]} on {last_used!r}" 3017 ) 3018 return dims

ValueError: conflicting sizes for dimension 'foo': length 1 on <this-array> and length 3 on {'lat': 'lat', 'lon': 'lon', 'time': 'time', 'foo': 'foo'} ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Is your feature request related to a problem?

Currently we need to know the order of dims when using region in to_zarr. Generally in xarray we're fine with the order, because we have the names, so this is a bit of an aberration. It means that code needs to carry around the correct order of dims.

Here's an MCVE:

```python

ds = xr.tutorial.load_dataset('air_temperature')

ds.to_zarr('foo', mode='w')

ds.transpose(..., 'lat').to_zarr('foo', mode='r+')

ValueError: variable 'air' already exists with different dimension names ('time', 'lat', 'lon') != ('time', 'lon', 'lat'), but changing variable dimensions is not supported by to_zarr().

```

Describe the solution you'd like

I think we should be able to transpose them based on the target?

Describe alternatives you've considered

No response

Additional context

No response

What is your issue?

I've been doing some work recently on our old friend numbagg, improving the ewm routines & adding some more.

I'm keen to get numbagg back in shape, doing the things that it does best, and trimming anything it doesn't. I notice that it has grouped calcs. Am I correct to think that flox does this better? I haven't been up with the latest. flox looks like it's particularly focused on dask arrays, whereas numpy_groupies, one of the inspirations for this, was applicable to numpy arrays too.

At least from the xarray perspective, are we OK to deprecate these numbagg functions, and direct folks to flox?

What is your issue?

Not the greatest issue facing the universe — but for the cause of consistency — should .reset_encoding be .drop_encoding, since it drops all encoding attributes?

For comparison: - .reset_coords — "Given names of coordinates, reset them to become variables." - '.drop_vars` — "Drop variables from this dataset."

Also ref #8258

Is your feature request related to a problem?

Currently we allow:

python da.where(lambda x: x.foo == 5)

...but we don't allow:

python da.where(lambda x: x.foo == 5, lambda x: x - x.shift(1))

...which would be nice

Describe the solution you'd like

No response

Describe alternatives you've considered

I don't think this offers many downsides — it's not like we want to fill the array with a callable object.

Additional context

No response

Generally this isn't a problem, since the coords are carried over by the resulting DataArrays:

``` python In [11]:

ds = xray.Dataset({ 'a':pd.DataFrame(pd.np.random.rand(10,3)), 'b':pd.Series(pd.np.random.rand(10)) }) ds.coords['c'] = pd.Series(pd.np.random.rand(10)) ds Out[11]: <xray.Dataset> Dimensions: (dim_0: 10, dim_1: 3) Coordinates: * dim_0 (dim_0) int64 0 1 2 3 4 5 6 7 8 9 * dim_1 (dim_1) int64 0 1 2 c (dim_0) float64 0.9318 0.2899 0.3853 0.6235 0.9436 0.7928 ... Data variables: a (dim_0, dim_1) float64 0.5707 0.9485 0.3541 0.5987 0.406 0.7992 ... b (dim_0) float64 0.4106 0.2316 0.5804 0.6393 0.5715 0.6463 ... In [12]:

ds.apply(lambda x: x*2) Out[12]: <xray.Dataset> Dimensions: (dim_0: 10, dim_1: 3) Coordinates: c (dim_0) float64 0.9318 0.2899 0.3853 0.6235 0.9436 0.7928 ... * dim_0 (dim_0) int64 0 1 2 3 4 5 6 7 8 9 * dim_1 (dim_1) int64 0 1 2 Data variables: a (dim_0, dim_1) float64 1.141 1.897 0.7081 1.197 0.812 1.598 ... b (dim_0) float64 0.8212 0.4631 1.161 1.279 1.143 1.293 0.3507 ... ```

But if there's an operation that removes the coords from the DataArrays, the coords are not there on the result (notice c below). Should the Dataset retain them? Either always or with a keep_coords argument, similar to keep_attrs.

``` python In [13]:

ds = xray.Dataset({ 'a':pd.DataFrame(pd.np.random.rand(10,3)), 'b':pd.Series(pd.np.random.rand(10)) }) ds.coords['c'] = pd.Series(pd.np.random.rand(10)) ds Out[13]: <xray.Dataset> Dimensions: (dim_0: 10, dim_1: 3) Coordinates: * dim_0 (dim_0) int64 0 1 2 3 4 5 6 7 8 9 * dim_1 (dim_1) int64 0 1 2 c (dim_0) float64 0.4121 0.2507 0.6326 0.4031 0.6169 0.441 0.1146 ... Data variables: a (dim_0, dim_1) float64 0.4813 0.2479 0.5158 0.2787 0.06672 ... b (dim_0) float64 0.2638 0.5788 0.6591 0.7174 0.3645 0.5655 ... In [14]:

ds.apply(lambda x: x.to_pandas()*2) Out[14]: <xray.Dataset> Dimensions: (dim_0: 10, dim_1: 3) Coordinates: * dim_0 (dim_0) int64 0 1 2 3 4 5 6 7 8 9 * dim_1 (dim_1) int64 0 1 2 Data variables: a (dim_0, dim_1) float64 0.9627 0.4957 1.032 0.5574 0.1334 0.8289 ... b (dim_0) float64 0.5275 1.158 1.318 1.435 0.7291 1.131 0.1903 ... ```

We had the main branch passing for a while, but unfortunately another test failure. Now in our new Linux py38-backend-api-v2 test case, intest_save_mfdataset_compute_false_roundtrip

link

``` self = <xarray.tests.test_backends.TestDask object at 0x7f821a0d6190>

def test_save_mfdataset_compute_false_roundtrip(self):
    from dask.delayed import Delayed

    original = Dataset({"foo": ("x", np.random.randn(10))}).chunk()
    datasets = [original.isel(x=slice(5)), original.isel(x=slice(5, 10))]
    with create_tmp_file(allow_cleanup_failure=ON_WINDOWS) as tmp1:
        with create_tmp_file(allow_cleanup_failure=ON_WINDOWS) as tmp2:
            delayed_obj = save_mfdataset(
                datasets, [tmp1, tmp2], engine=self.engine, compute=False
            )
            assert isinstance(delayed_obj, Delayed)
            delayed_obj.compute()
            with open_mfdataset(
                [tmp1, tmp2], combine="nested", concat_dim="x"
            ) as actual:

              assert_identical(actual, original)

E AssertionError: Left and right Dataset objects are not identical E
E
E Differing data variables: E L foo (x) float64 dask.array<chunksize=(5,), meta=np.ndarray> E R foo (x) float64 dask.array<chunksize=(10,), meta=np.ndarray>

/home/vsts/work/1/s/xarray/tests/test_backends.py:3274: AssertionError

AssertionError: Left and right Dataset objects are not identical

Differing data variables: L foo (x) float64 dask.array<chunksize=(5,), meta=np.ndarray> R foo (x) float64 dask.array<chunksize=(10,), meta=np.ndarray> ```

@aurghs & @alexamici — are you familiar with this? Thanks in advance

What is your issue?

Hi team — apologies but I mistakenly pushed to main. Less than a minute later I pushed another commit reverting the commit.

I'll check my git shortcuts tomorrow to ensure this can't happen by default — I thought I had the push remote set correctly, but possibly something doesn't use that.

Would we consider adding protection from pushes to main? (Am not blaming that setting tbc...)

(Will close this issue)

What happened?

We lose the time coord here — Dimensions without coordinates: time:

```python ds = xr.tutorial.load_dataset("air_temperature") ds.rolling_exp(time=5).mean()

<xarray.Dataset> Dimensions: (lat: 25, time: 2920, lon: 53) Coordinates: * lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0 * lon (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0 Dimensions without coordinates: time Data variables: air (time, lat, lon) float32 241.2 242.5 243.5 ... 296.4 296.1 295.7 ```

(I realize I wrote this, I didn't think this used to happen, but either it always did or I didn't write good enough tests... mea culpa)

What did you expect to happen?

We keep the time coords, like we do for normal rolling:

python In [2]: ds.rolling(time=5).mean() Out[2]: <xarray.Dataset> Dimensions: (lat: 25, lon: 53, time: 2920) Coordinates: * lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0 * lon (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0 * time (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00

Minimal Complete Verifiable Example

Python (as above)

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

What is your issue?

It looks like the doc build is failing on a pandas docstring:

``` /home/docs/checkouts/readthedocs.org/user_builds/xray/conda/8156/lib/python3.10/site-packages/cartopy/io/init.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/50m_physical/ne_50m_coastline.zip warnings.warn(f'Downloading: {url}', DownloadWarning) reading sources... [ 99%] user-guide/reshaping reading sources... [ 99%] user-guide/terminology reading sources... [ 99%] user-guide/time-series reading sources... [ 99%] user-guide/weather-climate reading sources... [100%] whats-new

/home/docs/checkouts/readthedocs.org/user_builds/xray/conda/8156/lib/python3.10/site-packages/pandas/core/indexes/base.py:docstring of pandas.core.indexes.base.Index.join:14: WARNING: Inline literal start-string without end-string. /home/docs/checkouts/readthedocs.org/user_builds/xray/conda/8156/lib/python3.10/site-packages/pandas/core/indexes/base.py:docstring of pandas.core.indexes.base.Index.join:15: WARNING: Inline literal start-string without end-string. looking for now-outdated files... none found ```

(also including the cartopy warning in case that's relevant)

Is this expected? Is anyone familiar enough with the doc build to know whether we can disable warnings from 3rd party modules?

What is your issue?

Hi team! I've been out of the flow for a bit. I'm just starting to add a couple of small changes to the library, and I'm having mypy issues I've spent 20 mins trying to figure out without much success.

When I run mypy locally, I get a lot of errors. That's even on the most recent commit which passed in CI, which suggests it's on me rather than Xarray... I've downgraded mypy versions to 1.4.1, which was the version that passed there.

But I can't figure out what it is — does anyone else have this? We can close if it's just me...

Is your feature request related to a problem?

I've been following https://github.com/arogozhnikov/einops with interest, and thought it would be worth raising whether we could offer an xarray integration, eventually inspired by the reshape discussion at https://github.com/pydata/xarray/discussions/7217#discussioncomment-4150613. (I thought there might have been a discussion / issue already, but couldn't find one).

Einops offers a string query to do common array operations. There's a good intro at https://github.com/arogozhnikov/einops/blob/master/docs/1-einops-basics.ipynb

The

Describe the solution you'd like

Because our dimension names are full names, it wouldn't be quite as terse, and it has lots of overlap with our existing operations. But I find string queries are often easy to understand, particularly for those who have less experience with xarray but are familiar with einsum & frends [^1].

Applying einops to xarray

The einops example, (no xarray yet):

python rearrange(ims, 'b h w c -> h (b w) c')

This could likely be something like, noting that we don't need the string before the ->, as our names are already defined:

python rearrange(ims, '-> height (batch width) color')

...or, if we wanted to name the new dimension rather than take an implicit dimension name like batch_width, at the cost of new einops syntax:

python rearrange(ims, '-> height width=(batch width) color')

...or if we wanted a method on a DataArray:

python ims.einops.rearrange('height width=(batch width) color')

Sometimes we would want the lvalue; e.g. to unstack:

python rearrange(ims, 'batch (height height2) color -> height (batch width) color', height2=2)

...and the same principles apply for reductions such as mean, for example

python ims.einops.reduce('-> height width color', 'mean')

...would be equivalent to:

```python ims.mean('batch')

or

ims.groupby(['height', 'width', 'color']).mean(...) ```

[^1]: I frequently find myself using .query, or pd.eval, also string queries, and these are great with tools that take user input and folks can pass a string

Describe alternatives you've considered

No response

Additional context

This probably needs a champion to drive, and realistically that's probably not me for quite a while. But if we can get consensus that this would be interesting, and someone is up for doing it, I think this could be a v cool feature.

I'll also tag @arogozhnikov for info

What happened?

Currently if someone supplies a path that doesn't exist, we get quite a long stack trace, without really saying that the path doesn't exist.

What did you expect to happen?

Possibly a FileNotFoundError

Minimal Complete Verifiable Example

Python xr.open_zarr('x.zarr')

Relevant log output

```Python In [1]: xr.open_zarr('x.zarr') <ipython-input-1-8be4b98d9b20>:1: RuntimeWarning: Failed to open Zarr store with consolidated metadata, falling back to try reading non-consolidated metadata. This is typically much slower for opening a dataset. To silence this warning, consider:

1. Consolidating metadata in this existing store with zarr.consolidate_metadata().
2. Explicitly setting consolidated=False, to avoid trying to read consolidate metadata, or
3. Explicitly setting consolidated=True, to raise an error in this case instead of falling back to try reading non-consolidated metadata. xr.open_zarr('x.zarr')

KeyError Traceback (most recent call last) ~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/xarray/backends/zarr.py in open_group(cls, store, mode, synchronizer, group, consolidated, consolidate_on_close, chunk_store, storage_options, append_dim, write_region, safe_chunks, stacklevel) 347 try: --> 348 zarr_group = zarr.open_consolidated(store, **open_kwargs) 349 except KeyError:

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/zarr/convenience.py in open_consolidated(store, metadata_key, mode, **kwargs) 1186 # setup metadata store -> 1187 meta_store = ConsolidatedMetadataStore(store, metadata_key=metadata_key) 1188

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/zarr/storage.py in init(self, store, metadata_key) 2643 # retrieve consolidated metadata -> 2644 meta = json_loads(store[metadata_key]) 2645

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/zarr/storage.py in getitem(self, key) 894 else: --> 895 raise KeyError(key) 896

KeyError: '.zmetadata'

During handling of the above exception, another exception occurred:

GroupNotFoundError Traceback (most recent call last) <ipython-input-1-8be4b98d9b20> in <cell line: 1>() ----> 1 xr.open_zarr('x.zarr')

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/xarray/backends/zarr.py in open_zarr(store, group, synchronizer, chunks, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, consolidated, overwrite_encoded_chunks, chunk_store, storage_options, decode_timedelta, use_cftime, **kwargs) 750 } 751 --> 752 ds = open_dataset( 753 filename_or_obj=store, 754 group=group,

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, backend_kwargs, args, *kwargs) 493 494 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None) --> 495 backend_ds = backend.open_dataset( 496 filename_or_obj, 497 drop_variables=drop_variables,

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/xarray/backends/zarr.py in open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, synchronizer, consolidated, chunk_store, storage_options, stacklevel) 798 799 filename_or_obj = _normalize_path(filename_or_obj) --> 800 store = ZarrStore.open_group( 801 filename_or_obj, 802 group=group,

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/xarray/backends/zarr.py in open_group(cls, store, mode, synchronizer, group, consolidated, consolidate_on_close, chunk_store, storage_options, append_dim, write_region, safe_chunks, stacklevel) 363 stacklevel=stacklevel, 364 ) --> 365 zarr_group = zarr.open_group(store, **open_kwargs) 366 elif consolidated: 367 # TODO: an option to pass the metadata_key keyword

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/zarr/hierarchy.py in open_group(store, mode, cache_attrs, synchronizer, path, chunk_store, storage_options) 1180 if contains_array(store, path=path): 1181 raise ContainsArrayError(path) -> 1182 raise GroupNotFoundError(path) 1183 1184 elif mode == 'w':

GroupNotFoundError: group not found at path '' ```

Anything else we need to know?

No response

Environment

INSTALLED VERSIONS

commit: None python: 3.9.12 (main, Mar 26 2022, 15:44:31) [Clang 13.1.6 (clang-1316.0.21.2)] python-bits: 64 OS: Darwin OS-release: 21.3.0 machine: arm64 processor: arm byteorder: little LC_ALL: en_US.UTF-8 LANG: None LOCALE: ('en_US', 'UTF-8') libhdf5: None libnetcdf: None

xarray: 2022.3.0 pandas: 1.4.1 numpy: 1.22.3 scipy: None netCDF4: None pydap: None h5netcdf: None h5py: None Nio: None zarr: 2.11.1 cftime: None nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: 2021.12.0 distributed: 2021.12.0 matplotlib: None cartopy: None seaborn: None numbagg: None fsspec: 2021.11.1 cupy: None pint: None sparse: None setuptools: 60.9.3 pip: 21.3.1 conda: None pytest: 6.2.5 IPython: 7.32.0 sphinx: None

What is your issue?

I can't work out what is causing this, and can't repro locally, though I've tried to ensure the same things are installed.

The bisect is: - Passes: https://github.com/pydata/xarray/runs/6233389985?check_suite_focus=true - Fails: https://github.com/pydata/xarray/runs/6237267544?check_suite_focus=true

Probably we have to skip it in the meantime, which is a shame

Is there a better way of locking the dependency versions so we can rule that out? I generally don't use conda, and poetry is great at this. Is there a conda equivalent?

What is your issue?

From https://github.com/pydata/xarray/pull/6472, and some PRs this weekend:

Is anyone familiar with what's going on with RTD? Did our concurrency limit drop?

Are there alternatives (e.g. running the tests on GHA even if the actual docs get built on RTD?). If we have to pay RTD for a subscription for a bit until we make changes then we could do that (I'm happy to given my recently poor contribution track-record!)

What is your issue?

@pydata/xarray

Because our pipeline takes a while, it can be helpful to have an option to "merge when tests pass" — I've now set that up. So you can click here and it'll do just that.

Someone annoyingly / confusingly, the "required checks" need to be specified manually, in https://github.com/pydata/xarray/settings/branch_protection_rules/2465574 — there's no option for just "all checks".

So if we change the checks — e.g. add Python 3.11 — that list needs to be updated. If we remove a check from the our CI and don't update the list, it won't be possible to merge the PR without clicking the red "Admin Override" box — so we should keep it up to date.

What happened?

I'm testing adding an issue with a <details> section; ref https://github.com/pydata/xarray/pull/6486

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

Relevant log output

No response

Anything else we need to know?

No response

Environment

What happened:

We have catestrophic performance on the html repr of some long multi-indexed data arrays. Here's a case of it taking 12s.

Minimal Complete Verifiable Example:

```python import xarray as xr

ds = xr.tutorial.load_dataset("air_temperature") da = ds["air"].stack(z=[...])

da.shape

(3869000,)

%timeit -n 1 -r 1 da.repr_html()

12.4 s !!

```

Anything else we need to know?:

I thought we'd fixed some issues here: https://github.com/pydata/xarray/pull/4846/files

Environment:

Minimal Complete Verifiable Example:

```python In [82]: da = xr.DataArray(data, dims=list("ab")).stack(c=[...])

In [83]: data = np.random.RandomState(0).randn(1000, 500)

In [84]: da = xr.DataArray(data, dims=list("ab")).stack(c=[...])

In [85]: da.indexes['c'] Out[85]: MultiIndex([( 0, 0), ( 0, 1), ( 0, 2), ( 0, 3), ( 0, 4), ( 0, 5), ( 0, 6), ( 0, 7), ( 0, 8), ( 0, 9), ... (999, 490), (999, 491), (999, 492), (999, 493), (999, 494), (999, 495), (999, 496), (999, 497), (999, 498), (999, 499)], names=['a', 'b'], length=500000)

In [89]: da.indexes['c'].name ```

What you expected to happen:

In [89]: da.indexes['c'].name Out [89]: 'c'

Environment:

I just went to put an issue in with pre-commit, and it looks good:

Here are the docs: https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/syntax-for-githubs-form-schema

We could replace our "paste this value between in this markdown code block" with boxes that encapsulate this

Is your feature request related to a problem?

Currently https://github.com/pydata/xarray/blob/main/HOW_TO_RELEASE.md has a few steps around the stable branch

Describe the solution you'd like

In our dev call, we discussed the possibility of using main in place of stable and removing the stable branch

IIRC there's something we can do on RTD to make that replacement. (If anyone knows to hand, comment here; otherwise I can search for it).

Is there anything else we need to do apart from RTD?

Describe alternatives you've considered

No response

Additional context

No response

Currently we have two bots which comment on issues, I'd propose we remove them both - The "Unit Test Results". I don't find this that useful — do others? I thought it would be listing the tests that failed (I had thought it might even be doing duration changes etc), but it seems to just list the number of tests that failed, which isn't that useful. Or are we not using it correctly? - pep8speaks — I think this would be dominated by pre-commit.ci

I'd propose we add: - pre-commit.ci. This autofixes PRs! e.g. here's a test PR with a black error from pytest-accept: https://github.com/max-sixty/pytest-accept/pull/24. This could also replace the pre-commit GHA (though it's no great cost to duplicate it)

What happened:

Calling ds.to_dask_dataframe(set_index='lat') raises on attempting to create a MultiIndex.

What you expected to happen:

Shouldn't this create a normal index with just lat?

Minimal Complete Verifiable Example:

```python In [1]: ds = xr.tutorial.load_dataset('air_temperature')

In [2]: ds.to_dask_dataframe(set_index='lat')

NotImplementedError Traceback (most recent call last) <ipython-input-2-e13a093182d0> in <module> ----> 1 ds.to_dask_dataframe(set_index='lat')

~/workspace/xarray/xarray/core/dataset.py in to_dask_dataframe(self, dim_order, set_index) 5534 # triggers an error about multi-indexes, even if only one 5535 # dimension is passed -> 5536 df = df.set_index(dim_order) 5537 5538 return df

~/.asdf/installs/python/3.8.10/lib/python3.8/site-packages/dask/dataframe/core.py in set_index(failed resolving arguments) 4177 from .shuffle import set_index 4178 -> 4179 return set_index( 4180 self, 4181 other,

~/.asdf/installs/python/3.8.10/lib/python3.8/site-packages/dask/dataframe/shuffle.py in set_index(df, index, npartitions, shuffle, compute, drop, upsample, divisions, partition_size, **kwargs) 140 index = index[0] 141 else: --> 142 raise NotImplementedError( 143 "Dask dataframe does not yet support multi-indexes.\n" 144 "You tried to index with this index: %s\n"

NotImplementedError: Dask dataframe does not yet support multi-indexes. You tried to index with this index: ['lat', 'time', 'lon'] Indexes must be single columns only. ```

Anything else we need to know?:

Environment:

What happened:

We have a failure related to a dask release, I think. Here's a job that failed: https://github.com/pydata/xarray/pull/5365/checks?check_run_id=2757459587

It's the test: xarray/tests/test_computation.py::test_vectorize_dask_dtype_meta

```

    References
    ----------
    .. [1] https://docs.scipy.org/doc/numpy/reference/ufuncs.html
    .. [2] https://docs.scipy.org/doc/numpy/reference/c-api/generalized-ufuncs.html
    """
    # Input processing:
    ## Signature
    if not isinstance(signature, str):
        raise TypeError("`signature` has to be of type string")
    input_coredimss, output_coredimss = _parse_gufunc_signature(signature)

    ## Determine nout: nout = None for functions of one direct return; nout = int for return tuples
    nout = None if not isinstance(output_coredimss, list) else len(output_coredimss)

    ## Consolidate onto `meta`
    if meta is not None and output_dtypes is not None:

      raise ValueError(
            "Only one of `meta` and `output_dtypes` should be given (`meta` is preferred)."
        )

E ValueError: Only one of meta and output_dtypes should be given (meta is preferred).

```

Should we xfail this? Does anyone have thoughts for a quick fix?

Is your feature request related to a problem? Please describe.

Currently we test on python 3.7, 3.8, and 3.9; across Linux, Windows, and Mac.

Describe the solution you'd like

Is there any reason to test 3.8 on every commit? Is there any code that would work on 3.7 & 3.9 but not on 3.8?

It's no great cost, but getting the pipeline to be faster has (fairly distributed, often unseen) benefits.

Describe alternatives you've considered

• I'm not sure whether it's possible to query CI history, and see where failures have shown up? i.e. "which tests, if you never included them, would lead to no change in whether all the tests pass"?
• We could run these as "confirmation" tests on main only
• We could maybe also cut out Mac & Windows 3.7? Less confident about these.

I'm about to release this as v0.18.2: https://github.com/pydata/xarray/compare/v0.18.1...max-sixty:release-0.18.2?expand=1 given https://github.com/pydata/xarray/issues/5346

Let me know any thoughts @pydata/xarray , thanks

What happened:

The new docs are looking great! One small issue — the lists don't appear as lists; e.g.

from https://xarray.pydata.org/en/latest/generated/xarray.Dataset.query.html

Do we need to change the rst convention?

What you expected to happen:

As bullets, with linebreaks

Is your feature request related to a problem? Please describe. Currently .clip can either take a scalar or a identically dimensioned array. numpy allows for broadcasting, which we can do in xarray too:

```python

In [11]: da = xr.DataArray(np.arange(24).reshape(2,4,3), dims=['a','b','c']) ...: da Out[11]: <xarray.DataArray (a: 2, b: 4, c: 3)> array([[[ 0, 1, 2], [ 3, 4, 5], [ 6, 7, 8], [ 9, 10, 11]],

   [[12, 13, 14],
    [15, 16, 17],
    [18, 19, 20],
    [21, 22, 23]]])

Dimensions without coordinates: a, b, c

In [12]: da.clip(da.mean('a'))

ValueError Traceback (most recent call last) <ipython-input-12-c8b9c6d0e0a1> in <module> ----> 1 da.clip(da.mean('a'))

[...]

~/.asdf/installs/python/3.8.8/lib/python3.8/site-packages/xarray/core/computation.py in apply_variable_ufunc(func, signature, exclude_dims, dask, output_dtypes, vectorize, keep_attrs, dask_gufunc_kwargs, *args) 741 data = as_compatible_data(data) 742 if data.ndim != len(dims): --> 743 raise ValueError( 744 "applied function returned data with unexpected " 745 f"number of dimensions. Received {data.ndim} dimension(s) but "

ValueError: applied function returned data with unexpected number of dimensions. Received 3 dimension(s) but expected 2 dimensions with names: ('b', 'c') ```

Adding in a broadcast_like currently allows it to work, but we can do the equivalent internally:

```python In [20]: da.clip(da.mean('a').broadcast_like(da)) Out[20]: <xarray.DataArray (a: 2, b: 4, c: 3)> array([[[ 6., 7., 8.], [ 9., 10., 11.], [12., 13., 14.], [15., 16., 17.]],

   [[12., 13., 14.],
    [15., 16., 17.],
    [18., 19., 20.],
    [21., 22., 23.]]])

Dimensions without coordinates: a, b, c ```

Additional context Numpy broadcasts the argument: https://numpy.org/doc/stable/reference/generated/numpy.clip.html

Is your feature request related to a problem? Please describe.

Currently xarray objects containting datetimes don't operate with pandas' offset objects:

python times = pd.date_range("2000-01-01", freq="6H", periods=10) ds = xr.Dataset( { "foo": (["time", "x", "y"], np.random.randn(10, 5, 3)), "bar": ("time", np.random.randn(10), {"meta": "data"}), "time": times, } ) ds.attrs["dsmeta"] = "dsdata" ds.resample(time="24H").mean("time").time + to_offset("8H")

raises: ```

TypeError Traceback (most recent call last) <ipython-input-29-f9de46fe6c54> in <module> ----> 1 ds.resample(time="24H").mean("time").time + to_offset("8H")

/usr/local/lib/python3.8/site-packages/xarray/core/dataarray.py in func(self, other) 2763 2764 variable = ( -> 2765 f(self.variable, other_variable) 2766 if not reflexive 2767 else f(other_variable, self.variable)

/usr/local/lib/python3.8/site-packages/xarray/core/variable.py in func(self, other) 2128 with np.errstate(all="ignore"): 2129 new_data = ( -> 2130 f(self_data, other_data) 2131 if not reflexive 2132 else f(other_data, self_data)

TypeError: unsupported operand type(s) for +: 'numpy.ndarray' and 'pandas._libs.tslibs.offsets.Hour' ```

This is an issue because pandas resampling has deprecated loffset — from our test suite:

``` xarray/tests/test_dataset.py::TestDataset::test_resample_loffset /Users/maximilian/workspace/xarray/xarray/tests/test_dataset.py:3844: FutureWarning: 'loffset' in .resample() and in Grouper() is deprecated.

df.resample(freq="3s", loffset="8H")

becomes:

from pandas.tseries.frequencies import to_offset df = df.resample(freq="3s").mean() df.index = df.index.to_timestamp() + to_offset("8H")

ds.bar.to_series().resample("24H", loffset="-12H").mean()

```

...and so we'll need to support something like this in order to maintain existing behavior.

Describe the solution you'd like I'm not completely sure; I think probably supporting the operations between xarray objects containing datetime objects and pandas' offset objects.

Numpy warns when computing over an all-NaN slice. We handle that case reasonably and so should handle and discard the warning.

MCVE Code Sample

```python In [1]: import xarray as xr

In [2]: import numpy as np

In [3]: da = xr.DataArray(np.asarray([np.nan]*3))

In [4]: da
Out[4]: <xarray.DataArray (dim_0: 3)> array([nan, nan, nan]) Dimensions without coordinates: dim_0

In [6]: da.mean()
[...]/python3.6/site-packages/xarray/core/nanops.py:142: RuntimeWarning: Mean of empty slice return np.nanmean(a, axis=axis, dtype=dtype) Out[6]: <xarray.DataArray ()> array(nan)

```

Expected Output

No warning

Problem Description

Somewhat discussed in https://github.com/pydata/xarray/issues/1164, and https://github.com/pydata/xarray/issues/1652, but starting a separate issue as it's more important than just noise in the test suite, and not covered by the existing work on comparisons & arithmetic

Output of xr.show_versions()

What happened:

The _repr_html_ method of large arrays seems very slow — 4.78s in the case of a 100m value array; and the general repr seems fairly slow — 1.87s. Here's a quick example. I haven't yet investigated how dependent it is on there being a MultiIndex.

What you expected to happen:

We should really focus on having good repr performance, given how essential it is to any REPL workflow.

Minimal Complete Verifiable Example:

```python In [10]: import xarray as xr ...: import numpy as np ...: import pandas as pd

In [11]: idx = pd.MultiIndex.from_product([range(10_000), range(10_000)])

In [12]: df = pd.DataFrame(range(100_000_000), index=idx)

In [13]: da = xr.DataArray(df)

In [14]: da Out[14]: <xarray.DataArray (dim_0: 100000000, dim_1: 1)> array([[ 0], [ 1], [ 2], ..., [99999997], [99999998], [99999999]]) Coordinates: * dim_0 (dim_0) MultiIndex - dim_0_level_0 (dim_0) int64 0 0 0 0 0 0 0 ... 9999 9999 9999 9999 9999 9999 - dim_0_level_1 (dim_0) int64 0 1 2 3 4 5 6 ... 9994 9995 9996 9997 9998 9999 * dim_1 (dim_1) int64 0

In [26]: %timeit repr(da) 1.87 s ± 7.33 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [27]: %timeit da.repr_html() 4.78 s ± 1.8 s per loop (mean ± std. dev. of 7 runs, 1 loop each) ```

Environment:

Currently almost none of the docstrings pass running pytest --doctest-modules xarray/core, though mostly for easy reasons.

Should we run these in CI?

I've recently started using docstring tests in another project, and they've work pretty well.

CC @keewis

I'm happy to debug this further but before I do, is this an issue people have seen before? I'm running tests on master and hit an issue very early on.

FWIW I don't use netCDF, and don't think I've got that installed

Code Sample, a copy-pastable example if possible

```python ========================================================================== FAILURES ========================================================================== _________ ScipyInMemoryDataTest.test_bytesio_pickle __________

self = <xarray.tests.test_backends.ScipyInMemoryDataTest testMethod=test_bytesio_pickle>

@pytest.mark.skipif(PY2, reason='cannot pickle BytesIO on Python 2')
def test_bytesio_pickle(self):
    data = Dataset({'foo': ('x', [1, 2, 3])})
    fobj = BytesIO(data.to_netcdf())
    with open_dataset(fobj, autoclose=self.autoclose) as ds:

      unpickled = pickle.loads(pickle.dumps(ds))

E TypeError: can't pickle _thread.lock objects

xarray/tests/test_backends.py:1384: TypeError ```

Problem description

[this should explain why the current behavior is a problem and why the expected output is a better solution.]

Expected Output

Skip or pass backends tests

Output of xr.show_versions()

Is your feature request related to a problem? Please describe. Could cov & corr ignore missing values?

Describe the solution you'd like Currently any NaN in an dimension over which cov / corr is calculated gives a NaN result:

```python In [1]: import xarray as xr ...: import numpy as np ...: da = xr.DataArray([[1, 2], [1, np.nan]], dims=["x", "time"]) ...: da Out[1]: <xarray.DataArray (x: 2, time: 2)> array([[ 1., 2.], [ 1., nan]]) Dimensions without coordinates: x, time

In [2]: xr.cov(da,da) Out[2]: <xarray.DataArray ()> array(nan) ```

That's explained here as: python # 4. Compute covariance along the given dim # N.B. `skipna=False` is required or there is a bug when computing # auto-covariance. E.g. Try xr.cov(da,da) for # da = xr.DataArray([[1, 2], [1, np.nan]], dims=["x", "time"]) cov = (demeaned_da_a * demeaned_da_b).sum(dim=dim, skipna=False) / (valid_count)

Without having thought about it for too long, I'm not sure I understand this, and couldn't find any discussion in the PR. Adding this diff seems to fail tests around NaN values but no others:

``diff diff --git a/xarray/core/computation.py b/xarray/core/computation.py index 1f2a8a8e..1fc95fe1 100644 --- a/xarray/core/computation.py +++ b/xarray/core/computation.py @@ -1256,7 +1256,8 @@ def _cov_corr(da_a, da_b, dim=None, ddof=0, method=None): # N.B.skipna=False` is required or there is a bug when computing # auto-covariance. E.g. Try xr.cov(da,da) for # da = xr.DataArray([[1, 2], [1, np.nan]], dims=["x", "time"]) - cov = (demeaned_da_a * demeaned_da_b).sum(dim=dim, skipna=False) / (valid_count) + cov = (demeaned_da_a * demeaned_da_b).sum(dim=dim, skipna=True, min_count=1) / (valid_count) + # cov = (demeaned_da_a * demeaned_da_b).sum(dim=dim, skipna=False) / (valid_count)

 if method == "cov":
     return cov

```

Does anyone know off-hand the logic here?

Follow up to https://github.com/pydata/xarray/pull/2174

Pasting from https://github.com/pydata/xarray/pull/2174#issuecomment-392111566

• [x] stack
• [x] shift
• [x] roll
• [x] set_index
• [x] reorder_levels
• [x] rolling
• [ ] resample (not yet, we still support old behavior for the first positional arguments with a warning)

...potentially rename (I often trip myself up on that)?

Is this a mistake or am I missing something?

http://xray.readthedocs.org/en/stable/generated/xarray.DataArray.count.html?highlight=count#xarray.DataArray.count

skipna : bool, optional If True, skip missing values (as marked by NaN). By default, only skips missing values for float dtypes; other dtypes either do not have a sentinel missing value (int) or skipna=True has not been implemented (object, datetime64 or timedelta64).

I was seeing some slow performance around to_xarray() on MultiIndexed series, and found that unstacking one of the dimensions before running to_xarray(), and then restacking with to_array() was ~30x faster. This time difference is consistent with larger data sizes.

To reproduce:

Create a series with a MultiIndex, ensuring the MultiIndex isn't a simple product:

```python s = pd.Series( np.random.rand(100000), index=pd.MultiIndex.from_product([ list('abcdefhijk'), list('abcdefhijk'), pd.DatetimeIndex(start='2000-01-01', periods=1000, freq='B'), ]))

cropped = s[::3] cropped.index=pd.MultiIndex.from_tuples(cropped.index, names=list('xyz'))

cropped.head()

x y z

a a 2000-01-03 0.993989

2000-01-06 0.850518

2000-01-11 0.068944

2000-01-14 0.237197

2000-01-19 0.784254

dtype: float64

```

Two approaches for getting this into xarray; 1 - Simple .to_xarray():

```python

current_method = cropped.to_xarray()

<xarray.DataArray (x: 10, y: 10, z: 1000)> array([[[0.993989, nan, ..., nan, 0.721663], [ nan, nan, ..., 0.58224 , nan], ..., [ nan, 0.369382, ..., nan, nan], [0.98558 , nan, ..., nan, 0.403732]],

   [[     nan,      nan, ..., 0.493711,      nan],
    [     nan, 0.126761, ...,      nan,      nan],
    ...,
    [0.976758,      nan, ...,      nan, 0.816612],
    [     nan,      nan, ..., 0.982128,      nan]],

   ...,

   [[     nan, 0.971525, ...,      nan,      nan],
    [0.146774,      nan, ...,      nan, 0.419806],
    ...,
    [     nan,      nan, ..., 0.700764,      nan],
    [     nan, 0.502058, ...,      nan,      nan]],

   [[0.246768,      nan, ...,      nan, 0.079266],
    [     nan,      nan, ..., 0.802297,      nan],
    ...,
    [     nan, 0.636698, ...,      nan,      nan],
    [0.025195,      nan, ...,      nan, 0.629305]]])

Coordinates: * x (x) object 'a' 'b' 'c' 'd' 'e' 'f' 'h' 'i' 'j' 'k' * y (y) object 'a' 'b' 'c' 'd' 'e' 'f' 'h' 'i' 'j' 'k' * z (z) datetime64[ns] 2000-01-03 2000-01-04 ... 2003-10-30 2003-10-31 ```

This takes 536 ms

2 - unstack in pandas first, and then use to_array to do the equivalent of a restack: proposed_version = ( cropped .unstack('y') .to_xarray() .to_array('y') )

This takes 17.3 ms

To confirm these are identical:

``` proposed_version_adj = ( proposed_version .assign_coords(y=proposed_version['y'].astype(object)) .transpose(*current_version.dims) )

proposed_version_adj.equals(current_version)

True

```

Problem description

A default operation is much slower than a (potentially) equivalent operation that's not the default.

I need to look more at what's causing the issues. I think it's to do with the .reindex(full_idx), but I'm unclear why it's so much faster in the alternative route, and whether there's a fix that we can make to make the default path fast.

Output of xr.show_versions()

MCVE Code Sample

I thought we might have had an issue (and maybe solved it?) but couldn't find it anywhere. Forgive me if I'm duplicating.

```python xr.DataArray(np.random.rand(100,5,1))

<xarray.DataArray (dim_0: 100, dim_1: 5, dim_2: 1)> array([[[0.71333665], [0.93820892], [0.48678056], [0.07299961], [0.63542414]],

*** Deleted 400 lines ***

   [[0.29987457],
    [0.55963998],
    [0.25976744],
    [0.80062955],
    [0.503025  ]],

   [[0.48255097],
    [0.55861315],
    [0.36059861],
    [0.96539665],
    [0.05674621]],

   [[0.81389941],
    [0.55745028],
    [0.20348983],
    [0.63390148],
    [0.94698865]],

   [[0.16792246],
    [0.9252646 ],
    [0.38596734],
    [0.17168077],
    [0.18162088]],

   [[0.04526339],
    [0.70028912],
    [0.72388995],
    [0.97481276],
    [0.66155381]],

   [[0.15058745],
    [0.57646963],
    [0.53382085],
    [0.24696459],
    [0.77601528]],

   [[0.6752243 ],
    [0.84991466],
    [0.87758404],
    [0.70828751],
    [0.04033709]]])

Dimensions without coordinates: dim_0, dim_1, dim_2 ```

Expected Output

With larger arrays, it's much more reasonable:

``` <xarray.DataArray (dim_0: 500, dim_1: 6, dim_2: 1)> array([[[0.9680447 ], [0.12554914], [0.9163406 ], [0.63710986], [0.97778361], [0.6419909 ]],

   [[0.48480678],
    [0.31214637],
    [0.72270997],
    [0.81523543],
    [0.34327902],
    [0.80941523]],

   [[0.92192284],
    [0.47841933],
    [0.00760903],
    [0.83886152],
    [0.88538772],
    [0.6532889 ]],

   ...,

   [[0.39558324],
    [0.42220218],
    [0.56731915],
    [0.27388751],
    [0.51097741],
    [0.62824705]],

   [[0.97379019],
    [0.0311196 ],
    [0.09790975],
    [0.65206508],
    [0.14369363],
    [0.09683937]],

   [[0.71318171],
    [0.88591664],
    [0.30032286],
    [0.97324135],
    [0.10250702],
    [0.03973667]]])

Dimensions without coordinates: dim_0, dim_1, dim_2 ```

Problem Description

Something like 40 lines is probably a reasonable place to truncate?

Output of xr.show_versions()

```python

import xarray as xr import pandas as pd da = xr.DataArray(pd.Series(1, pd.period_range('2000-1', '2000-12', freq='W')).rename_axis('date'))

da.resample('B', 'date', 'ffill')

TypeError Traceback (most recent call last) <ipython-input-1-eb64a66a8d1f> in <module>() 3 da = xr.DataArray(pd.Series(1, pd.period_range('2000-1', '2000-12', freq='W')).rename_axis('date')) 4 ----> 5 da.resample('B', 'date', 'ffill')

/Users/maximilian/drive/workspace/xarray/xarray/core/common.py in resample(self, freq, dim, how, skipna, closed, label, base, keep_attrs) 577 time_grouper = pd.TimeGrouper(freq=freq, how=how, closed=closed, 578 label=label, base=base) --> 579 gb = self.groupby_cls(self, group, grouper=time_grouper) 580 if isinstance(how, basestring): 581 f = getattr(gb, how)

/Users/maximilian/drive/workspace/xarray/xarray/core/groupby.py in init(self, obj, group, squeeze, grouper, bins, cut_kwargs) 242 raise ValueError('index must be monotonic for resampling') 243 s = pd.Series(np.arange(index.size), index) --> 244 first_items = s.groupby(grouper).first() 245 if first_items.isnull().any(): 246 full_index = first_items.index

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pandas/core/generic.py in groupby(self, by, axis, level, as_index, sort, group_keys, squeeze, kwargs) 3989 return groupby(self, by=by, axis=axis, level=level, as_index=as_index, 3990 sort=sort, group_keys=group_keys, squeeze=squeeze, -> 3991 kwargs) 3992 3993 def asfreq(self, freq, method=None, how=None, normalize=False):

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pandas/core/groupby.py in groupby(obj, by, kwds) 1509 raise TypeError('invalid type: %s' % type(obj)) 1510 -> 1511 return klass(obj, by, kwds) 1512 1513

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pandas/core/groupby.py in init(self, obj, keys, axis, level, grouper, exclusions, selection, as_index, sort, group_keys, squeeze, **kwargs) 368 level=level, 369 sort=sort, --> 370 mutated=self.mutated) 371 372 self.obj = obj

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pandas/core/groupby.py in _get_grouper(obj, key, axis, level, sort, mutated) 2390 # a passed-in Grouper, directly convert 2391 if isinstance(key, Grouper): -> 2392 binner, grouper, obj = key._get_grouper(obj) 2393 if key.key is None: 2394 return grouper, [], obj

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pandas/tseries/resample.py in _get_grouper(self, obj) 1059 def _get_grouper(self, obj): 1060 # create the resampler and return our binner -> 1061 r = self._get_resampler(obj) 1062 r._set_binner() 1063 return r.binner, r.grouper, r.obj

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pandas/tseries/resample.py in _get_resampler(self, obj, kind) 1055 raise TypeError("Only valid with DatetimeIndex, " 1056 "TimedeltaIndex or PeriodIndex, " -> 1057 "but got an instance of %r" % type(ax).name) 1058 1059 def _get_grouper(self, obj):

TypeError: Only valid with DatetimeIndex, TimedeltaIndex or PeriodIndex, but got an instance of 'Index' ```

MCVE Code Sample

We have a very verbose warning in 0.15: it prints on every groupby on an object with multidimensional coords.

So the notebook I'm currently working on has red sections like: /home/mroos/.local/lib/python3.7/site-packages/xarray/core/common.py:664: FutureWarning: This DataArray contains multi-dimensional coordinates. In the future, the dimension order of these coordinates will be restored as well unless you specify restore_coord_dims=False. self, group, squeeze=squeeze, restore_coord_dims=restore_coord_dims /home/mroos/.local/lib/python3.7/site-packages/xarray/core/common.py:664: FutureWarning: This DataArray contains multi-dimensional coordinates. In the future, the dimension order of these coordinates will be restored as well unless you specify restore_coord_dims=False. self, group, squeeze=squeeze, restore_coord_dims=restore_coord_dims /home/mroos/.local/lib/python3.7/site-packages/xarray/core/common.py:664: FutureWarning: This DataArray contains multi-dimensional coordinates. In the future, the dimension order of these coordinates will be restored as well unless you specify restore_coord_dims=False. self, group, squeeze=squeeze, restore_coord_dims=restore_coord_dims /home/mroos/.local/lib/python3.7/site-packages/xarray/core/common.py:664: FutureWarning: This DataArray contains multi-dimensional coordinates. In the future, the dimension order of these coordinates will be restored as well unless you specify restore_coord_dims=False. self, group, squeeze=squeeze, restore_coord_dims=restore_coord_dims /home/mroos/.local/lib/python3.7/site-packages/xarray/core/common.py:664: FutureWarning: This DataArray contains multi-dimensional coordinates. In the future, the dimension order of these coordinates will be restored as well unless you specify restore_coord_dims=False. self, group, squeeze=squeeze, restore_coord_dims=restore_coord_dims

Unless there's a way of reducing its verbosity (e.g. only print once per session?), let's aim to push the change through and remove the warning soon?

```python

Your code here

In [2]: import xarray as xr

In [4]: import numpy as np

In [16]: da = xr.DataArray(np.random.rand(2,3), dims=list('ab'))

In [17]: da = da.assign_coords(foo=(('a','b'),np.random.rand(2,3)))

In [18]: da.groupby('a').mean(...)
[...]/python3.6/site-packages/xarray/core/common.py:664: FutureWarning: This DataArray contains multi-dimensional coordinates. In the future, the dimension order of these coordinates will be restored as well unless you specify restore_coord_dims=False. self, group, squeeze=squeeze, restore_coord_dims=restore_coord_dims Out[18]: <xarray.DataArray (a: 2)> array([0.59216558, 0.58616892]) Dimensions without coordinates: a

```

Output of xr.show_versions()

This is out of date: http://xarray.pydata.org/en/stable/roadmap.html#current-core-developers

MCVE Code Sample

```python xr.DataArray(coords=dict(a=' '.join(['hello world' for _ in range(100)])))

<xarray.DataArray ()> array(nan) Coordinates: a <U5999 'hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world hello world' ```

Expected Output

<xarray.DataArray ()> array(nan) Coordinates: a <U5999 'hello world ... hello world'

Problem Description

I think mostly the same as https://github.com/pydata/xarray/issues/1319 but for coords

Output of xr.show_versions()

I've seen a couple of failures recently on this test. Flaky tests are really annoying and would be great to fix or if impossible, remove it. Does anyone have any ideas what's causing this?

``` __ TestPseudoNetCDFFormat.test_uamiv_format_write __

self = <xarray.tests.test_backends.TestPseudoNetCDFFormat object at 0x7f15352b9d00>

def test_uamiv_format_write(self):
    fmtkw = {"format": "uamiv"}

    expected = open_example_dataset(
        "example.uamiv", engine="pseudonetcdf", backend_kwargs=fmtkw
    )
    with self.roundtrip(
        expected,
        save_kwargs=fmtkw,
        open_kwargs={"backend_kwargs": fmtkw},
        allow_cleanup_failure=True,
    ) as actual:

      assert_identical(expected, actual)

E AssertionError: Left and right Dataset objects are not identical E
E
E
E Differing attributes: E L WTIME: 190117 E R WTIME: 190118

xarray/tests/test_backends.py:3563: AssertionError ```

MCVE Code Sample

```python

In a notebook with html repr enabled

xr.Dataset({0: (('a','b'), np.random.rand(2,3))})

gives:

AttributeError Traceback (most recent call last) /j/office/app/research-python/conda/envs/2019.10/lib/python3.7/site-packages/IPython/core/formatters.py in call(self, obj) 343 method = get_real_method(obj, self.print_method) 344 if method is not None: --> 345 return method() 346 return None 347 else:

~/.local/lib/python3.7/site-packages/xarray/core/dataset.py in repr_html(self) 1632 if OPTIONS["display_style"] == "text": 1633 return f"

{escape(repr(self))}

~/.local/lib/python3.7/site-packages/xarray/core/formatting_html.py in dataset_repr(ds) 268 dim_section(ds), 269 coord_section(ds.coords), --> 270 datavar_section(ds.data_vars), 271 attr_section(ds.attrs), 272 ]

~/.local/lib/python3.7/site-packages/xarray/core/formatting_html.py in _mapping_section(mapping, name, details_func, max_items_collapse, enabled) 165 return collapsible_section( 166 name, --> 167 details=details_func(mapping), 168 n_items=n_items, 169 enabled=enabled,

~/.local/lib/python3.7/site-packages/xarray/core/formatting_html.py in summarize_vars(variables) 131 vars_li = "".join( 132 f"

~/.local/lib/python3.7/site-packages/xarray/core/formatting_html.py in <genexpr>(.0) 131 vars_li = "".join( 132 f"

~/.local/lib/python3.7/site-packages/xarray/core/formatting_html.py in summarize_variable(name, var, is_index, dtype, preview) 96 cssclass_idx = " class='xr-has-index'" if is_index else "" 97 dims_str = f"({', '.join(escape(dim) for dim in var.dims)})" ---> 98 name = escape(name) 99 dtype = dtype or escape(str(var.dtype)) 100

/j/office/app/research-python/conda/envs/2019.10/lib/python3.7/html/init.py in escape(s, quote) 17 translated. 18 """ ---> 19 s = s.replace("&", "&") # Must be done first! 20 s = s.replace("<", "<") 21 s = s.replace(">", ">")

AttributeError: 'int' object has no attribute 'replace'

<xarray.Dataset> Dimensions: (a: 2, b: 3) Dimensions without coordinates: a, b Data variables: 0 (a, b) float64 0.5327 0.927 0.8582 0.8825 0.9478 0.09475 ```

Problem Description

I think this may be an uncomplicated fix: coerce the keys to str

Output of xr.show_versions()

Panel is removed from pandas (as a result of the success of xarray! :grin: ), but we're still attempting to call it from .to_pandas()

MCVE Code Sample

```python

Your code here

In [4]: import numpy as np

In [1]: import xarray as xr
In [5]: xr.DataArray(np.random.rand(2,3,4)).to_pandas()

TypeError Traceback (most recent call last) <ipython-input-5-7d1b667d5cac> in <module> ----> 1 xr.DataArray(np.random.rand(2,3,4)).to_pandas()

~/workspace/corpfin/.venv/lib64/python3.6/site-packages/xarray/core/dataarray.py in to_pandas(self) 2267 ) 2268 indexes = [self.get_index(dim) for dim in self.dims] -> 2269 return constructor(self.values, *indexes) 2270 2271 def to_dataframe(self, name: Hashable = None) -> pd.DataFrame:

TypeError: object() takes no parameters

```

Expected Output

Either a MultiIndexed DataFrame or a proper error (4D gives ValueError: cannot convert arrays with 4 dimensions into pandas objects)

Output of xr.show_versions()

Branching from https://github.com/pydata/xarray/issues/3770

Here's the proposal: allow lambdas on methods where the primary argument is a single xarray object, and interpret lambas as though they'd be supplied in a pipe method followed by the current method. Taking the example from the linked issue:

```python

In [1]: import xarray as xr

In [2]: import numpy as np

In [3]: da = xr.DataArray(np.random.rand(2,3))

In [4]: da.where(da > 0.5)
Out[4]: <xarray.DataArray (dim_0: 2, dim_1: 3)> array([[ nan, 0.71442406, nan], [0.55748705, nan, nan]]) Dimensions without coordinates: dim_0, dim_1

this should be equivalent (currently not valid)

In [5]: da.where(lambda x: x > 0.5)

the longer version (currently works)

In [5]: da.pipe(lambda x: x.where(x > 0.5))

```

Others I miss from pandas: assign, and loc. I haven't gone through the list though assume there are others; we don't have to agree 100% on the list before starting with the most obvious ones, assuming we're in agreement with the principle.

Could we add the ability to use an ellipsis to represent all dims in more places? For example, stack:

MCVE Code Sample

```python

In [14]: data = np.arange(15, 301, 15).reshape(2, 10)
...: da = xr.DataArray(data, dims=('y', 'x'), attrs={'test': 'test'})
...:

In [15]: da.stack(z=['x','y'])
Out[15]: <xarray.DataArray (z: 20)> array([ 15, 165, 30, 180, 45, 195, 60, 210, 75, 225, 90, 240, 105, 255, 120, 270, 135, 285, 150, 300]) Coordinates: * z (z) MultiIndex - x (z) int64 0 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 - y (z) int64 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 Attributes: test: test

In [16]: da.stack(z=[...])

KeyError Traceback (most recent call last) <ipython-input-16-a92d0ffe931a> in <module> ----> 1 da.stack(z=[...])

~/workspace/./.venv/lib64/python3.6/site-packages/xarray/core/dataarray.py in stack(self, dimensions, dimensions_kwargs) 1739 DataArray.unstack 1740 """ -> 1741 ds = self._to_temp_dataset().stack(dimensions, dimensions_kwargs) 1742 return self._from_temp_dataset(ds) 1743

~/workspace/./.venv/lib64/python3.6/site-packages/xarray/core/dataset.py in stack(self, dimensions, **dimensions_kwargs) 3291 result = self 3292 for new_dim, dims in dimensions.items(): -> 3293 result = result._stack_once(dims, new_dim) 3294 return result 3295

~/workspace/./.venv/lib64/python3.6/site-packages/xarray/core/dataset.py in _stack_once(self, dims, new_dim) 3246 3247 # consider dropping levels that are unused? -> 3248 levels = [self.get_index(dim) for dim in dims] 3249 idx = utils.multiindex_from_product_levels(levels, names=dims) 3250 variables[new_dim] = IndexVariable(new_dim, idx)

~/workspace/./.venv/lib64/python3.6/site-packages/xarray/core/dataset.py in <listcomp>(.0) 3246 3247 # consider dropping levels that are unused? -> 3248 levels = [self.get_index(dim) for dim in dims] 3249 idx = utils.multiindex_from_product_levels(levels, names=dims) 3250 variables[new_dim] = IndexVariable(new_dim, idx)

~/workspace/./.venv/lib64/python3.6/site-packages/xarray/core/common.py in get_index(self, key) 378 """ 379 if key not in self.dims: --> 380 raise KeyError(key) 381 382 try:

KeyError: Ellipsis

```

Expected Output

Identical between ... and listing all dimensions

Output of xr.show_versions()

MCVE Code Sample

I optimistically tried passing a lambda to where* to avoid another .pipe. To my surprise it ran! But unfortunately it just ignored the lambda, as though all values were True.

```python

In [1]: import xarray as xr

In [2]: import numpy as np

In [3]: da = xr.DataArray(np.random.rand(2,3))

In [4]: da.where(da > 0.5)
Out[4]: <xarray.DataArray (dim_0: 2, dim_1: 3)> array([[ nan, 0.71442406, nan], [0.55748705, nan, nan]]) Dimensions without coordinates: dim_0, dim_1

this should fail!

In [5]: da.where(lambda x: x > 0.5)
Out[5]: <xarray.DataArray (dim_0: 2, dim_1: 3)> array([[0.26085668, 0.71442406, 0.05816167], [0.55748705, 0.15293073, 0.12766453]]) Dimensions without coordinates: dim_0, dim_1 ```

Expected Output

Raise a TypeError when passed the lambda

* (maybe that could work, but separate discussion)

Output of xr.show_versions()

We are moving some code over from pandas to Xarray, and one of the biggest missing features is exponential functions, e.g. series.ewm(span=20).mean().

It looks like we can write these as gufuncs without too much trouble in numba. But I also notice that numbagg hasn't changed in a while and that we chose bottleneck for many of the functions in Xarray.

• Is numba a good approach for these?
• As well as our own internal use, could we add numba functions to Xarray, or are there dependency issues?
• Tangentially, I'd be interested why we're using bottleneck rather than numbagg for the existing functions

MCVE Code Sample

```python In [1]: import xarray as xr

In [3]: import numpy as np

In [4]: ds = xr.Dataset( ...: {"foo": (("x", "y"), np.random.rand(4, 3))}, ...: coords={"x": [10, 20, 30, 40], "letters": ("x", list("abba"))}, ...: )

In [5]: ds.groupby('letters')
Out[5]: DatasetGroupBy, grouped over 'letters' 2 groups with labels 'a', 'b'.

In [8]: ds.groupby('letters').sum(...) / ds.groupby('letters').count(...)
Out[8]: <xarray.Dataset> Dimensions: (letters: 2) Coordinates: * letters (letters) object 'a' 'b' Data variables: foo (letters) float64 0.4182 0.4995

In [9]: ds.groupby('letters').pipe(lambda x: x.sum() / x.count())

AttributeError Traceback (most recent call last) <ipython-input-9-c9b142ea051b> in <module> ----> 1 ds.groupby('letters').pipe(lambda x: x.sum() / x.count())

AttributeError: 'DatasetGroupBy' object has no attribute 'pipe'

```

Expected Output

I think we could add groupby.pipe, as a convenience?

Output of xr.show_versions()

``` mypy .

xarray/core/common.py:337: error: Too few arguments for "init_subclass" xarray/core/dataset.py:398: error: Too few arguments for "init_subclass" xarray/core/dataset.py:873: error: Incompatible default for argument "attrs" (default has type "object", argument has type "Optional[Dict[Hashable, Any]]") xarray/core/dataset.py:874: error: Incompatible default for argument "indexes" (default has type "object", argument has type "Optional[Dict[Any, Any]]") xarray/core/dataset.py:875: error: Incompatible default for argument "encoding" (default has type "object", argument has type "Optional[Dict[Any, Any]]") xarray/core/dataset.py:922: error: Incompatible default for argument "attrs" (default has type "object", argument has type "Optional[Dict[Hashable, Any]]") xarray/core/dataset.py:923: error: Incompatible default for argument "indexes" (default has type "object", argument has type "Dict[Hashable, Any]") xarray/core/dataset.py:937: error: Incompatible default for argument "attrs" (default has type "object", argument has type "Dict[Hashable, Any]") xarray/core/dataarray.py:213: error: Too few arguments for "init_subclass" Found 9 errors in 3 files (checked 122 source files)

```

mypy --version mypy 0.740

@crusaderky had a good idea to allow ellipsis (...) as a placeholder for 'other dims' in transpose.

What about using it as a placeholder for xr.ALL_DIMS in groupby etc operations? I find it nicer than custom sentinel values, and I think should be fairly low-confusion—thoughts?

I just saw https://numpy.org/neps/nep-0029-deprecation_policy.html, which suggests dropping 3.5 support earlier this year (though NEP is dated to this July)

Ofc, existing xarray versions would still work with 3.5, only upgrades would require the new version (which I know everyone knows, but reinforces we don't need to wait for everyone to move off 3.5)

What do we think about a minor release soon?

My colleague just hit https://github.com/pydata/xarray/pull/3145 which reminded me the current release can make filing an issue confusing for folks

We've had a v good pipeline of changes in just a month: https://github.com/pydata/xarray/blob/master/doc/whats-new.rst#L18

I'm happy to help with this; ref https://github.com/pydata/xarray/issues/2998#issuecomment-516218628

(edit: original failure was mistaken) Does anyone know off hand why this is failing?

ResolvePackageNotFound: - pandas=0.19 - python=3.5.0

Worst case we could drop it... https://github.com/pydata/xarray/issues/3293

https://dev.azure.com/xarray/xarray/_build/results?buildId=695

```python

=================================== FAILURES =================================== ___ TestSparseVariable.test_unary_op ___

self = <xarray.tests.test_sparse.TestSparseVariable object at 0x7f24f0b21b70>

def test_unary_op(self):

  sparse.utils.assert_eq(-self.var.data, -self.data)

E AttributeError: module 'sparse' has no attribute 'utils'

xarray/tests/test_sparse.py:285: AttributeError ___ TestSparseVariable.test_univariate_ufunc _____

self = <xarray.tests.test_sparse.TestSparseVariable object at 0x7f24ebc2bb38>

def test_univariate_ufunc(self):

  sparse.utils.assert_eq(np.sin(self.data), xu.sin(self.var).data)

E AttributeError: module 'sparse' has no attribute 'utils'

xarray/tests/test_sparse.py:290: AttributeError ___ TestSparseVariable.test_bivariate_ufunc ______

self = <xarray.tests.test_sparse.TestSparseVariable object at 0x7f24f02a7e10>

def test_bivariate_ufunc(self):

  sparse.utils.assert_eq(np.maximum(self.data, 0), xu.maximum(self.var, 0).data)

E AttributeError: module 'sparse' has no attribute 'utils'

xarray/tests/test_sparse.py:293: AttributeError ___ TestSparseVariable.testpickle ____

self = <xarray.tests.test_sparse.TestSparseVariable object at 0x7f24f04f2c50>

def test_pickle(self):
    v1 = self.var
    v2 = pickle.loads(pickle.dumps(v1))

  sparse.utils.assert_eq(v1.data, v2.data)

E AttributeError: module 'sparse' has no attribute 'utils'

xarray/tests/test_sparse.py:307: AttributeError ```

Any ideas?

Any ideas why __hdf5libversion__ wouldn't be available? Shall I put a try / except around it?

```python In [4]: import xarray as xr

In [5]: xr.show_versions()

ModuleNotFoundError Traceback (most recent call last) /usr/local/lib/python3.7/site-packages/xarray/util/print_versions.py in netcdf_and_hdf5_versions() 64 try: ---> 65 import netCDF4 66 libhdf5_version = netCDF4.hdf5libversion

ModuleNotFoundError: No module named 'netCDF4'

During handling of the above exception, another exception occurred:

AttributeError Traceback (most recent call last) <ipython-input-5-6f391305f2fe> in <module> ----> 1 xr.show_versions()

/usr/local/lib/python3.7/site-packages/xarray/util/print_versions.py in show_versions(file) 78 sys_info = get_sys_info() 79 ---> 80 sys_info.extend(netcdf_and_hdf5_versions()) 81 82 deps = [

/usr/local/lib/python3.7/site-packages/xarray/util/print_versions.py in netcdf_and_hdf5_versions() 69 try: 70 import h5py ---> 71 libhdf5_version = h5py.hdf5libversion 72 except ImportError: 73 pass

AttributeError: module 'h5py' has no attribute 'hdf5libversion' ```

I check I'm on the latest h5py: pip install h5py -U Thu Jul 18 16:47:29 2019 Requirement already up-to-date: h5py in ./Library/Python/3.7/lib/python/site-packages (2.9.0) Requirement already satisfied, skipping upgrade: numpy>=1.7 in /usr/local/lib/python3.7/site-packages (from h5py) (1.16.4) Requirement already satisfied, skipping upgrade: six in ./Library/Python/3.7/lib/python/site-packages (from h5py) (1.12.0)

ref https://github.com/pydata/xarray/pull/3090#issuecomment-510323490

Do we want the bot commenting on the PR, at least while the early checks are wrong? People can always click on Details in the Codecov check (e.g. https://codecov.io/gh/pydata/xarray/compare/8f0d9e5c9909c93a90306ed7cb5a80c1c2e1c97d...ab6960f623017afdc99c34bcbb69b402aea3f7d4/diff) to see a full report.

Happy to PR to disable, lmk

I realize this is a topic primed for bike-shedding, so I've held off suggesting, but here goes...

Is there any interest in a pipe operator to make xarray syntax easier? Our internal code has much code like:

```python

delinq_c = ( base_ds['delinq_comp_array'] # lots of pipe lines! .pipe(lambda x: x - base_ds['delinq_comp_array'].mean('fsym_id')) .pipe(lambda x: x / delinq_diff) .pipe(lambda x: x * 2) )

```

...with lots of new lines starting with .pipe. The fluent code is great for sequences of logic, but the requirement to have .pipe on each line adds verbosity.

The addition of a pipe operator would allow for:

```python

delinq_c = ( base_ds['delinq_comp_array'] >> lambda x: x - base_ds['delinq_comp_array'].mean('fsym_id') >> lambda x: x / delinq_diff >> lambda x: x * 2 )

```

This requires (ab)using an existing python operator, such as >> (bitshift operator) or | (or operator). Airflow uses >>, Beam (and bash) use |. While xarray doesn't allow either to be used, other libraries might incorrectly assume their presence implies they work consistently with those definitions.

Python has explicitly not added this, nor reduced the character count of lambda x:, so this would be somewhat a dissent from their standards, and introduce potential confusion to completely new xarray users.

I remember some discussions at pandas on similar topics. I can't find them all, but here's an issue re adding the X term https://github.com/pandas-dev/pandas/issues/13133

https://github.com/pydata/xarray/blob/master/xarray/core/dataset.py#L4085

Should that be called coord or dim? Currently there's a variable that's undefined:

python raise ValueError('Coordinate {} does not exist.'.format(dim)) I would have made a quick fix but not sure the correct name

Currently we use pycodestyle: https://github.com/pydata/xarray/blob/ccd0b047ea8ca89c68ab6cfa942557e676e7d402/.travis.yml#L63

I think we used to use flake8. I can't find / remember the reason we moved to pycodestyle.

master has some non-trivial issues that flake would catch, including a test overwritting another and undefined variables: ``` flake8 xarray --ignore=I,W503,W504,F401,E265,E402

xarray/core/options.py:62:8: F632 use ==/!= to compare str, bytes, and int literals xarray/core/dataset.py:4148:69: F821 undefined name 'dim' xarray/backends/netCDF4_.py:177:12: F632 use ==/!= to compare str, bytes, and int literals xarray/tests/test_dataarray.py:1264:9: F841 local variable 'foo' is assigned to but never used xarray/tests/test_dataarray.py:1270:18: F821 undefined name 'x' xarray/tests/test_dataarray.py:1301:5: F811 redefinition of unused 'test_reindex_fill_value' from line 1262 xarray/tests/test_dataarray.py:1647:16: F632 use ==/!= to compare str, bytes, and int literals xarray/tests/test_dataarray.py:1648:16: F632 use ==/!= to compare str, bytes, and int literals xarray/tests/test_dataset.py:4759:8: F632 use ==/!= to compare str, bytes, and int literals xarray/tests/test_dataset.py:4761:10: F632 use ==/!= to compare str, bytes, and int literals xarray/tests/test_distributed.py:62:9: F811 redefinition of unused 'loop' from line 12 xarray/tests/test_distributed.py:92:9: F811 redefinition of unused 'loop' from line 12 xarray/tests/test_distributed.py:117:49: F811 redefinition of unused 'loop' from line 12 xarray/tests/test_distributed.py:141:53: F811 redefinition of unused 'loop' from line 12 xarray/tests/test_distributed.py:152:51: F811 redefinition of unused 'loop' from line 12 ```

Happy to fix these in a PR. For ensuring these don't crop up again, any objection to flake8?

```python In [15]: xr.DataArray(np.random.randint(2,size=(100,100)).astype(bool)).shift(dim_0=0) Out[15]: <xarray.DataArray (dim_0: 100, dim_1: 100)> array([[False, True, True, ..., True, True, False], [False, True, False, ..., False, True, True], [False, True, False, ..., False, True, False], ..., [False, True, False, ..., False, True, True], [True, False, True, ..., False, False, False], [False, True, True, ..., True, True, False]], dtype=object) # <-- could be bool Dimensions without coordinates: dim_0, dim_1

```

Problem description

This causes memory bloat

Expected Output

As above with dtype=bool

Output of xr.show_versions()

The shift=0 is mainly theoretical. To avoid casting to object in practical scenarios, we could add a fill_value argument (e.g. fill_value=False) and fill with that rather than NaN

CC @Ivocrnkovic

This is extremely minor, I generally wouldn't report it.

We're using classes inherited from Dataset more & more - this works really well for classes with a lot of array-like properties that can be aligned, and allows @property to lazily compute some calcs. (any feedback on this approach very welcome)

The top of the repr is incorrect

python <xarray.SecurityMeasure> Dimensions: (date: 6647, security: 285, sub_measure: 4) Coordinates: ...

Could just be the qualified name.

``` python In [1]: arr = xr.DataArray(np.random.rand(4, 3), ...: ...: [('time', pd.date_range('2000-01-01', periods=4)), ...: ...: ('space', ['IA', 'IL', 'IN'])]) ...: ...:

In [17]: arr.coords['space2'] = ('space', ['A','B','C'])

In [18]: arr Out[18]: <xarray.DataArray (time: 4, space: 3)> array([[ 0.05187049, 0.04743067, 0.90329666], [ 0.59482538, 0.71014366, 0.86588207], [ 0.51893157, 0.49442107, 0.10697737], [ 0.16068189, 0.60756757, 0.31935279]]) Coordinates: * time (time) datetime64[ns] 2000-01-01 2000-01-02 2000-01-03 2000-01-04 * space (space) |S2 'IA' 'IL' 'IN' space2 (space) |S1 'A' 'B' 'C' ```

Now try to select on the space2 coord:

``` python In [19]: arr.sel(space2='A')

ValueError Traceback (most recent call last) <ipython-input-19-eae5e4b64758> in <module>() ----> 1 arr.sel(space2='A')

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/xarray/core/dataarray.pyc in sel(self, method, tolerance, indexers) 601 """ 602 return self.isel(indexing.remap_label_indexers( --> 603 self, indexers, method=method, tolerance=tolerance)) 604 605 def isel_points(self, dim='points', **indexers):

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/xarray/core/dataarray.pyc in isel(self, indexers) 588 DataArray.sel 589 """ --> 590 ds = self._to_temp_dataset().isel(indexers) 591 return self._from_temp_dataset(ds) 592

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/xarray/core/dataset.pyc in isel(self, **indexers) 908 invalid = [k for k in indexers if k not in self.dims] 909 if invalid: --> 910 raise ValueError("dimensions %r do not exist" % invalid) 911 912 # all indexers should be int, slice or np.ndarrays

ValueError: dimensions ['space2'] do not exist ```

Is there an easier way to do this? I couldn't think of anything...

CC @justinkuosixty

As discussed with @jhamman , I've been attempting to manually close / follow up on stale issues & PRs, given we had 500+ issues

Any thoughts on doing this automatically with https://github.com/probot/stale?

This could ask for an affirmation after an issue has been uncommented for a year, and then close a month later if there's no response

master is breaking on both the docs and python3.6 print_versions: https://travis-ci.org/pydata/xarray/jobs/479834129

/home/travis/.travis/job_stages: line 104: 3514 Segmentation fault (core dumped) python xarray/util/print_versions.py The command "python xarray/util/print_versions.py" failed and exited with 139 during .

Has anyone seen this before? I can't replicate locally, but I likely don't have the same dependencies

I returned to do some work on xarray, and looks like there are lots of linting errors. Maybe we shouldn't worry ourselves with this; though let's make a deliberate decision

Is pep8speaks working well? One example is this PR: https://github.com/pydata/xarray/pull/2553, where it looks like pep8speaks initially complained, but doesn't have a test so was potentially lost in the discussion.

Any thoughts? I'm happy to help see if there are alternatives / pep8speaks can be added as a test / add a test in travis / fix these.

FWIW I've used black successfully in pandas-gbq and internally, but I think it's maybe too much change for a large project like xarray

./versioneer.py:943:-13592: W605 invalid escape sequence '\s' ./versioneer.py:943:-13408: W605 invalid escape sequence '\s' ./versioneer.py:943:-13228: W605 invalid escape sequence '\s' ./versioneer.py:943:-11196: W605 invalid escape sequence '\d' ./versioneer.py:943:-8162: W605 invalid escape sequence '\d' ./xarray/testing.py:148:1: W391 blank line at end of file ./xarray/core/options.py:66:80: E501 line too long (97 > 79 characters) ./xarray/core/combine.py:381:36: E226 missing whitespace around arithmetic operator ./xarray/core/combine.py:536:75: E211 whitespace before '(' ./xarray/core/combine.py:537:57: E126 continuation line over-indented for hanging indent ./xarray/core/dataarray.py:19:1: F401 '.options._get_keep_attrs' imported but unused ./xarray/core/dataset.py:15:1: F401 '.computation' imported but unused ./xarray/core/dataset.py:18:1: F401 '..conventions' imported but unused ./xarray/core/dataset.py:27:1: F401 '.dtypes.is_datetime_like' imported but unused ./xarray/core/dataset.py:32:1: F401 '.pycompat.integer_types' imported but unused ./xarray/core/dataset.py:520:5: E303 too many blank lines (2) ./xarray/core/dataset.py:739:80: E501 line too long (81 > 79 characters) ./xarray/core/dataset.py:750:80: E501 line too long (80 > 79 characters) ./xarray/core/dataset.py:768:80: E501 line too long (88 > 79 characters) ./xarray/core/dataset.py:779:80: E501 line too long (84 > 79 characters) ./xarray/core/dataset.py:788:80: E501 line too long (84 > 79 characters) ./xarray/core/dataset.py:795:80: E501 line too long (80 > 79 characters) ./xarray/core/dataset.py:811:80: E501 line too long (84 > 79 characters) ./xarray/core/dataset.py:828:80: E501 line too long (80 > 79 characters) ./xarray/core/dataset.py:1178:80: E501 line too long (92 > 79 characters) ./xarray/core/dataset.py:1365:13: F401 'dask' imported but unused ./xarray/core/dataset.py:1365:80: E501 line too long (92 > 79 characters) ./xarray/core/dataset.py:1624:80: E501 line too long (82 > 79 characters) ./xarray/core/dataset.py:1763:80: E501 line too long (82 > 79 characters) ./xarray/core/dataset.py:1866:80: E501 line too long (80 > 79 characters) ./xarray/core/dataset.py:1867:80: E501 line too long (80 > 79 characters) ./xarray/core/dataset.py:1977:16: E111 indentation is not a multiple of four ./xarray/core/dataset.py:1991:37: E126 continuation line over-indented for hanging indent ./xarray/core/dataset.py:2908:39: F812 list comprehension redefines 'dim' from line 2900 ./xarray/core/dataset.py:2976:80: E501 line too long (86 > 79 characters) ./xarray/core/dataset.py:2983:80: E501 line too long (82 > 79 characters) ./xarray/core/dataset.py:3027:80: E501 line too long (80 > 79 characters) ./xarray/core/dataset.py:3709:32: F812 list comprehension redefines 'dim' from line 3656 ./xarray/core/dataset.py:3749:80: E501 line too long (80 > 79 characters) ./xarray/core/dataset.py:3832:80: E501 line too long (83 > 79 characters) ./xarray/core/dataset.py:3912:80: E501 line too long (82 > 79 characters) ./xarray/core/dataset.py:3918:80: E501 line too long (84 > 79 characters) ./xarray/core/dataset.py:3929:80: E501 line too long (82 > 79 characters) ./xarray/core/dataset.py:3932:80: E501 line too long (88 > 79 characters) ./xarray/core/dataset.py:3933:80: E501 line too long (83 > 79 characters) ./xarray/core/common.py:3:1: F401 'warnings' imported but unused ./xarray/core/common.py:4:1: F401 'distutils.version.LooseVersion' imported but unused ./xarray/core/common.py:209:80: E501 line too long (90 > 79 characters) ./xarray/core/common.py:429:80: E501 line too long (82 > 79 characters) ./xarray/core/common.py:465:80: E501 line too long (86 > 79 characters) ./xarray/core/common.py:468:80: E501 line too long (84 > 79 characters) ./xarray/core/common.py:472:80: E501 line too long (81 > 79 characters) ./xarray/core/common.py:473:80: E501 line too long (87 > 79 characters) ./xarray/core/common.py:486:80: E501 line too long (82 > 79 characters) ./xarray/core/common.py:496:80: E501 line too long (80 > 79 characters) ./xarray/core/common.py:505:80: E501 line too long (80 > 79 characters) ./xarray/core/common.py:527:80: E501 line too long (84 > 79 characters) ./xarray/core/common.py:560:80: E501 line too long (80 > 79 characters) ./xarray/core/common.py:564:80: E501 line too long (81 > 79 characters) ./xarray/core/common.py:568:80: E501 line too long (85 > 79 characters) ./xarray/core/common.py:637:80: E501 line too long (81 > 79 characters) ./xarray/core/common.py:641:80: E501 line too long (85 > 79 characters) ./xarray/core/common.py:648:80: E501 line too long (86 > 79 characters) ./xarray/core/common.py:654:80: E501 line too long (86 > 79 characters) ./xarray/core/common.py:661:80: E501 line too long (89 > 79 characters) ./xarray/core/common.py:677:80: E501 line too long (80 > 79 characters) ./xarray/core/common.py:679:1: W293 blank line contains whitespace ./xarray/core/utils.py:511:13: W605 invalid escape sequence '\:' ./xarray/plot/plot.py:17:1: F401 'xarray.core.alignment.align' imported but unused ./xarray/plot/plot.py:259:5: E303 too many blank lines (2) ./xarray/backends/netcdf3.py:12:14: W605 invalid escape sequence '\(' ./xarray/backends/pynio_.py:12:1: I004 isort found an unexpected blank line in imports ./xarray/backends/api.py:18:1: I004 isort found an unexpected blank line in imports ./xarray/backends/api.py:26:9: F401 'netCDF4' imported but unused ./xarray/backends/api.py:30:13: F401 'pydap' imported but unused ./xarray/backends/api.py:41:9: F401 'Nio' imported but unused ./xarray/backends/api.py:46:9: F401 'cfgrib' imported but unused ./xarray/backends/api.py:59:9: F401 'scipy' imported but unused ./xarray/backends/api.py:68:9: F401 'netCDF4' imported but unused ./xarray/backends/api.py:72:13: F401 'scipy.io.netcdf' imported but unused ./xarray/backends/api.py:581:80: E501 line too long (80 > 79 characters) ./xarray/backends/api.py:648:80: E501 line too long (81 > 79 characters) ./xarray/backends/pseudonetcdf_.py:12:1: I004 isort found an unexpected blank line in imports ./xarray/backends/zarr.py:11:1: F401 '.common.ArrayWriter' imported but unused ./xarray/backends/zarr.py:240:80: E501 line too long (82 > 79 characters) ./xarray/backends/rasterio_.py:14:1: I004 isort found an unexpected blank line in imports ./xarray/backends/file_manager.py:6:1: I004 isort found an unexpected blank line in imports ./xarray/tests/test_combine.py:10:1: F401 'xarray.merge' imported but unused ./xarray/tests/test_combine.py:495:65: E211 whitespace before '(' ./xarray/tests/test_combine.py:523:52: E231 missing whitespace after ',' ./xarray/tests/test_combine.py:523:54: E231 missing whitespace after ',' ./xarray/tests/test_combine.py:523:61: E231 missing whitespace after ',' ./xarray/tests/test_combine.py:524:57: E241 multiple spaces after ',' ./xarray/tests/test_combine.py:525:55: E241 multiple spaces after ',' ./xarray/tests/test_combine.py:526:55: E241 multiple spaces after ',' ./xarray/tests/test_combine.py:527:57: E241 multiple spaces after ',' ./xarray/tests/test_combine.py:597:52: E203 whitespace before ',' ./xarray/tests/test_combine.py:646:80: E501 line too long (85 > 79 characters) ./xarray/tests/test_combine.py:647:80: E501 line too long (85 > 79 characters) ./xarray/tests/test_combine.py:658:80: E501 line too long (83 > 79 characters) ./xarray/tests/test_combine.py:659:80: E501 line too long (87 > 79 characters) ./xarray/tests/test_plot.py:1164:13: F841 local variable 'g' is assigned to but never used ./xarray/tests/test_utils.py:12:1: F401 'xarray.core.options.set_options' imported but unused ./xarray/tests/test_dataarray.py:13:1: F401 'xarray.set_options' imported but unused ./xarray/tests/test_backends.py:660:80: E501 line too long (82 > 79 characters) ./xarray/tests/test_backends.py:1345:9: F841 local variable 'zarr' is assigned to but never used ./xarray/tests/test_backends.py:2171:9: E303 too many blank lines (2) ./xarray/tests/test_backends.py:2197:36: E127 continuation line over-indented for visual indent ./xarray/tests/test_backends.py:2204:36: E126 continuation line over-indented for hanging indent ./xarray/tests/test_backends.py:2242:5: F811 redefinition of unused 'test_open_mfdataset' from line 2155 ./xarray/tests/test_backends.py:2965:80: E501 line too long (82 > 79 characters) ./xarray/tests/test_variable.py:30:1: I004 isort found an unexpected blank line in imports ./xarray/tests/test_distributed.py:3:1: F401 'distutils.version.LooseVersion' imported but unused ./xarray/tests/test_distributed.py:4:1: F401 'os' imported but unused ./xarray/tests/test_distributed.py:5:1: F401 'sys' imported but unused ./xarray/tests/test_distributed.py:7:1: F401 'tempfile' imported but unused ./xarray/tests/test_distributed.py:12:80: E501 line too long (81 > 79 characters) ./xarray/tests/test_distributed.py:14:1: F401 'dask.array' imported but unused ./xarray/tests/test_distributed.py:17:1: F401 'distributed.utils_test.loop' imported but unused ./xarray/tests/test_distributed.py:19:1: F401 'numpy as np' imported but unused ./xarray/tests/test_distributed.py:28:1: F401 '.raises_regex' imported but unused ./xarray/tests/test_distributed.py:66:1: F811 redefinition of unused 'loop' from line 17 ./xarray/tests/test_distributed.py:76:49: F841 local variable 'c' is assigned to but never used ./xarray/tests/test_distributed.py:96:1: F811 redefinition of unused 'loop' from line 17 ./xarray/tests/test_distributed.py:106:49: F841 local variable 'c' is assigned to but never used ./xarray/tests/test_distributed.py:121:1: E303 too many blank lines (3) ./xarray/tests/test_distributed.py:121:1: F811 redefinition of unused 'loop' from line 17 ./xarray/tests/test_distributed.py:126:9: F841 local variable 'zarr' is assigned to but never used ./xarray/tests/test_distributed.py:133:49: F841 local variable 'c' is assigned to but never used ./xarray/tests/test_distributed.py:147:1: F811 redefinition of unused 'loop' from line 17 ./xarray/tests/test_distributed.py:151:53: F841 local variable 'c' is assigned to but never used ./xarray/tests/test_distributed.py:158:1: F811 redefinition of unused 'loop' from line 17 ./xarray/tests/test_distributed.py:161:49: F841 local variable 'c' is assigned to but never used ./xarray/coding/cftime_offsets.py:422:17: W605 invalid escape sequence '\d' ./xarray/coding/cftimeindex.py:71:2: W605 invalid escape sequence '\-' ./xarray/coding/cftimeindex.py:71:2: W605 invalid escape sequence '\:' ./xarray/coding/cftimeindex.py:72:6: W605 invalid escape sequence '\d' ./xarray/coding/cftimeindex.py:72:6: W605 invalid escape sequence '\d' ./xarray/coding/cftimeindex.py:72:6: W605 invalid escape sequence '\d' ./xarray/coding/cftimeindex.py:72:6: W605 invalid escape sequence '\d' ./xarray/coding/cftimeindex.py:72:6: W605 invalid escape sequence '\d' ./xarray/coding/cftimeindex.py:72:6: W605 invalid escape sequence '\d'

```python

In [1]: import xarray as xr

In [3]: ds=xr.Dataset({'a':xr.DataArray([1,2,3])})

In [4]: xr.Dataset(ds) /Users/maximilian/drive/workspace/xarray/xarray/core/dataset.py:373: FutureWarning: iteration over an xarray.Dataset will change in xarray v0.11 to only include data variables, not coordinates. Iterate over the Dataset.variables property instead to preserve existing behavior in a forwards compatible manner. both_data_and_coords = [k for k in data_vars if k in coords] Out[4]: <xarray.Dataset> Dimensions: (dim_0: 3) Dimensions without coordinates: dim_0 Data variables: a (dim_0) int64 1 2 3 ```

Problem description

Currently we run both_data_and_coords = [k for k in data_vars if k in coords], where data_vars is the first arg to the constructor. So, when data_vars is a Dataset, we'll raise a warning.

More importantly: this raises the question of how we should handle xr.Dataset(ds) Currently we retain the coords, but discard attrs. That seems arbitrary? I'd have to check, but when we change __iter__ to only reference data_vars, we may also discard coords

I don't have a strong view. If you think that Foo(foo) should always return foo (which is often the case in pandas), then we should retain the full object. If you think that foo is supplied to the data_vars arg, then retaining only the data_vars seems reasonable

Expected Output

Output of xr.show_versions()

This has been a small-but-persistent issue for me for a while. I suspect that my perspective might be dependent on my current outlook, but socializing it here to test if it's secular...

Currently Dataset.keys() returns both variables and coordinates (but not its attrs keys):

python In [5]: ds=xr.Dataset({'a': (('x', 'y'), np.random.rand(10,2))}) In [12]: list(ds.keys()) Out[12]: ['a', 'x', 'y']

Is this conceptually correct? I would posit that a Dataset is a mapping of keys to variables, and the coordinates contain values that label that data.

So should Dataset.keys() instead return just the keys of the Variables?

We're often passing around a dataset as a Mapping of keys to values - but then when we run a function across each of the keys, we get something run on both the Variables' keys, and the Coordinate / label's keys.

In Pandas, DataFrame.keys() returns just the columns, so that conforms to what we need. While I think the xarray design is in general much better in these areas, this is one area that pandas seems to get correct - and because of the inconsistency between pandas & xarray, we're having to coerce our objects to pandas DataFrames before passing them off to functions that pull out their keys (this is also why we can't just look at ds.data_vars.keys() - because it breaks that duck-typing).

Does that make sense?

There's an elegant check against the xarray version to decide whether to run a test, so the test 'comes online' at 0.12: https://github.com/pydata/xarray/blob/638b251c622359b665208276a2cb23b0fbc5141b/xarray/tests/test_dataarray.py#L2029

But unfortunately, this seems very unstable in tests, because without a release, LooseVersion can't interpret the strings correctly (e.g. LooseVersion ('0.10.9+29.g33d9391a')) - A lot of the time it raises: https://travis-ci.org/max-sixty/xarray/jobs/437913418#L1036 - Occasionally it runs the test, failing to return that we're prior to 0.12: https://travis-ci.org/max-sixty/xarray/jobs/437914645#L5036

Here's the bug in the python issue tracker: https://bugs.python.org/issue14894

Is that synopsis correct? Should we attempt to take another approach? I'll disable it in my current check so tests can pass, but lmk thoughts.

If a coordinate is passed as a set, xr raises with a bad error message:

```python

In [12]: xr.Dataset(dict(date=[1,2,3], sec={4}))

MissingDimensionsError Traceback (most recent call last) <ipython-input-12-40ccdd94e21f> in <module>() ----> 1 xr.Dataset(dict(date=[1,2,3], sec={4}))

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/xarray/core/dataset.py in init(self, data_vars, coords, attrs, compat) 360 coords = {} 361 if data_vars is not None or coords is not None: --> 362 self._set_init_vars_and_dims(data_vars, coords, compat) 363 if attrs is not None: 364 self.attrs = attrs

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/xarray/core/dataset.py in _set_init_vars_and_dims(self, data_vars, coords, compat) 378 379 variables, coord_names, dims = merge_data_and_coords( --> 380 data_vars, coords, compat=compat) 381 382 self._variables = variables

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/xarray/core/merge.py in merge_data_and_coords(data, coords, compat, join) 363 objs = [data, coords] 364 explicit_coords = coords.keys() --> 365 return merge_core(objs, compat, join, explicit_coords=explicit_coords) 366 367

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/xarray/core/merge.py in merge_core(objs, compat, join, priority_arg, explicit_coords, indexes) 425 coerced = coerce_pandas_values(objs) 426 aligned = deep_align(coerced, join=join, copy=False, indexes=indexes) --> 427 expanded = expand_variable_dicts(aligned) 428 429 coord_names, noncoord_names = determine_coords(coerced)

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/xarray/core/merge.py in expand_variable_dicts(list_of_variable_dicts) 211 var_dicts.append(coords) 212 --> 213 var = as_variable(var, name=name) 214 sanitized_vars[name] = var 215

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/xarray/core/variable.py in as_variable(obj, name) 103 'cannot set variable %r with %r-dimensional data ' 104 'without explicit dimension names. Pass a tuple of ' --> 105 '(dims, data) instead.' % (name, data.ndim)) 106 obj = Variable(name, obj, fastpath=True) 107 else:

MissingDimensionsError: cannot set variable 'sec' with 0-dimensional data without explicit dimension names. Pass a tuple of (dims, data) instead. But OK if a list:python In [13]: xr.Dataset(dict(date=[1,2,3], sec=[4])) Out[13]: <xarray.Dataset> Dimensions: (date: 3, sec: 1) Coordinates: * date (date) int64 1 2 3 * sec (sec) int64 4 Data variables: empty

```

Problem description

There may be reasons to not allow sets: they're not ordered, so unless you're constructing your data using the coords, you'll get random results

The error message should be better though. And I would vote to handle sets the same as lists

Expected Output

Output of xr.show_versions()

I'd expect a to be a data_vars rather than a coord here:

python In [9]: xr.Dataset(data_vars={'a': [2,3]}, attrs={'name':'hello'}) Out[9]: <xarray.Dataset> Dimensions: (a: 2) Coordinates: * a (a) int64 2 3 Data variables: *empty* Attributes: name: hello

We have a DataArray unaligned to coords here:

``` python In [12]:

coords={'a': range(4), 'b': range(3)} da = xray.DataArray(np.random.rand(4,3), coords=coords) da Out[12]: <xray.DataArray (a: 4, b: 3)> array([[ 0.05126985, 0.95460352, 0.12853847], [ 0.20577943, 0.80265117, 0.46370886], [ 0.0226791 , 0.33068145, 0.55748573], [ 0.15943175, 0.20183347, 0.46907727]]) Coordinates: * a (a) int64 0 1 2 3 * b (b) int64 0 1 2 In [13]:

da_reindex = da.reindex(b=[1,2,0]) da_reindex Out[13]: <xray.DataArray (a: 4, b: 3)> array([[ 0.95460352, 0.12853847, 0.05126985], [ 0.80265117, 0.46370886, 0.20577943], [ 0.33068145, 0.55748573, 0.0226791 ], [ 0.20183347, 0.46907727, 0.15943175]]) Coordinates: * a (a) int64 0 1 2 3 * b (b) int64 1 2 0 ```

If we add this to Dataset and supply coords=coords, it raises, since there are conflicting coords:

``` python In [16]:

: ds = xray.Dataset(variables={'da':da_reindex}, coords=coords) ds

ValueError Traceback (most recent call last) <ipython-input-16-8e4afa10f781> in <module>() ----> 1 ds = xray.Dataset(variables={'da':da_reindex}, coords=coords) 2 ds

/usr/local/lib/python2.7/dist-packages/xray/core/dataset.pyc in init(self, variables, coords, attrs, compat) 317 coords = set() 318 if variables or coords: --> 319 self._set_init_vars_and_dims(variables, coords, compat) 320 if attrs is not None: 321 self.attrs = attrs

/usr/local/lib/python2.7/dist-packages/xray/core/dataset.pyc in _set_init_vars_and_dims(self, vars, coords, compat) 371 aligned = _align_variables(variables) 372 new_variables, new_coord_names = _expand_variables(aligned, --> 373 compat=compat) 374 375 new_coord_names.update(coords)

/usr/local/lib/python2.7/dist-packages/xray/core/dataset.pyc in _expand_variables(raw_variables, old_variables, compat) 142 add_variable(dim, coord.variable) 143 var = var.variable --> 144 add_variable(name, var) 145 146 return new_variables, new_coord_names

/usr/local/lib/python2.7/dist-packages/xray/core/dataset.pyc in add_variable(name, var) 130 raise ValueError('conflicting value for variable %s:\n' 131 'first value: %r\nsecond value: %r' --> 132 % (name, variables[name], var)) 133 if compat == 'broadcast_equals': 134 maybe_promote_or_replace(name, var)

ValueError: conflicting value for variable b: first value: <xray.Coordinate 'b' (b: 3)> array([1, 2, 0]) second value: <xray.Coordinate 'b' (b: 3)> array([0, 1, 2]) ```

But adding with __setitem__ aligns as expected:

``` python In [17]:

ds ds = xray.Dataset(coords=coords) ds['da']=da_reindex ds Out[17]: <xray.Dataset> Dimensions: (a: 4, b: 3) Coordinates: * a (a) int64 0 1 2 3 * b (b) int64 0 1 2 Data variables: da (a, b) float64 0.05127 0.9546 0.1285 0.2058 0.8027 0.4637 ... ```

Is this desired behavior? I could imagine aligning to coords if they are explicitly supplied, but I don't know the nuances here.

Based off http://www.rigtorp.se/2011/01/01/rolling-statistics-numpy.html, we wrote up a function that 'tricks' numpy into presenting an array that looks rolling, but without the O^2 memory requirements

Would people be interested in this going into xarray?

It seems to work really well on a few use-cases, but I imagine it's enough trickery that we might not want to support it in xarray. And, to be clear, it's strictly worse where we have rolling algos. But where we don't, you get a rolling apply without the python loops.

```python

def rolling_window_numpy(a, window): """ Make an array appear to be rolling, but using only a view http://www.rigtorp.se/2011/01/01/rolling-statistics-numpy.html """ shape = a.shape[:-1] + (a.shape[-1] - window + 1, window) strides = a.strides + (a.strides[-1],) return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)

def rolling_window(da, span, dim=None, new_dim='dim_0'): """ Adds a rolling dimension to a DataArray using only a view """ original_dims = da.dims da = da.transpose(*tuple(d for d in da.dims if d != dim) + (dim,))

result = apply_ufunc(
    rolling_window_numpy,
    da,
    output_core_dims=((new_dim,),),
    kwargs=(dict(window=span)))

return result.transpose(*(original_dims + (new_dim,)))

tests

import numpy as np import pandas as pd import pytest import xarray as xr

@pytest.fixture def da(dims): return xr.DataArray( np.random.rand(5, 10, 15), dims=(list('abc'))).transpose(*dims)

@pytest.fixture(params=[ list('abc'), list('bac'), list('cab'), ]) def dims(request): return request.param

def test_iterate_imputation_fills_missing(sample_data): sample_data.iloc[2, 2] = pd.np.nan result = iterate_imputation(sample_data) assert result.shape == sample_data.shape assert result.notnull().values.all()

def test_rolling_window(da, dims):

result = rolling_window(da, 3, dim='c', new_dim='x')

assert result.transpose(*list('abcx')).shape == (5, 10, 13, 3)

# should be a view, so doesn't have any larger strides
assert np.max(result.values.strides) == 10 * 15 * 8

def test_rolling_window_values():

da = xr.DataArray(np.arange(12).reshape(2, 6), dims=('item', 'date'))

rolling = rolling_window(da, 3, dim='date', new_dim='rolling_date')

expected = sum([11, 10, 9])
result = rolling.sum('rolling_date').isel(item=1, date=-1)
assert result == expected

```

I think with np.flip and bn.push, this should be simple. They're both fairly new and so would require version checks / upgrading the minimums.

One small issue, I wonder if anyone has come across this: bottleneck returns the numpy array rather than the DataArray - is that because it's not operating with the correct numpy interface?

Forward fill: array.values = bn.push(array.values, axis=array.get_axis_num(axis_name))

Backfill: ``` axis = array.get_axis_num(axis_name)

reverse for bfill

array = np.flip(array, axis=axis)

fill

array.values = bn.push(array.values, axis=axis)

reverse back to original

result = np.flip(scaling, axis=date_axis) ```

http://xray.readthedocs.org/en/stable/generated/xray.DataArray.count.html

name doesn't seem to be working correctly:

Dimension(s) over which to apply {name}.

Installing 3.6 environment (from the ci path in xarray): tests pass on master Then installing bottleneck: 3 tests in test_dataarray.py fail on master

I can debug further unless anyone has a view

Because xarray is all python and generally not doing much compute itself (i.e. it marshals other libraries to do that), this hasn't been that important.

IIRC most of the performance issues have arisen where xarray builds on (arguably) shaky foundations, like PeriodIndex.

Though as we mature, is it worth adding some benchmarks?

If so, what's a good way to do this? Pandas uses asv successfully. I don't have experience with https://github.com/ionelmc/pytest-benchmark but that could be a lower cost way of getting started. Any others?

Should .count() be available on .rolling objects?

```python

  sample_count = array.rolling(**windows).count() - 1.0

E AttributeError: 'DataArrayRolling' object has no attribute 'count' ```

This looks like it's available with dir / tab complete, but actually isn't:

``` python In [13]: xr.DataArray(np.random.rand(10,3)).to_dataset('dim_1').rolling

AttributeError Traceback (most recent call last) <ipython-input-13-438d3638a0d0> in <module>() ----> 1 xr.DataArray(np.random.rand(10,3)).to_dataset('dim_1').rolling

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/xarray/core/common.py in getattr(self, name) 135 return source[name] 136 raise AttributeError("%r object has no attribute %r" % --> 137 (type(self).name, name)) 138 139 def setattr(self, name, value):

AttributeError: 'Dataset' object has no attribute 'rolling' ```

I think this could be easy to implement as an .apply operation? (indeed, that could be a reasonable path for a whole host of operations - i.e. try and apply them to each array in the ds?)

Also, as a very narrow point, I'm not sure why .rolling_cls is public? Probably should be private?

Finally, the Rolling implementation is pretty sweet. I've been getting my hands dirty in the pandas one recently, and that we can have something as well featured as that with so few lines of code 👍

I need some guidance on how to handle this.

Background

PeriodIndex has a 'non-numpy' dtype now: ```python In [2]: i = pd.PeriodIndex(start=2000, freq='A', periods=10)

In [3]: i Out[3]: PeriodIndex(['2000', '2001', '2002', '2003', '2004', '2005', '2006', '2007', '2008', '2009'], dtype='period[A-DEC]', freq='A-DEC')

In [6]: i.dtype Out[6]: period[A-DEC] ```

When .values or .__array__() are called, the Periods are boxed, which is really slow. The underlying ints are stored in ._values:

```python In [25]: i.values Out[25]: array([Period('2000', 'A-DEC'), Period('2001', 'A-DEC'), Period('2002', 'A-DEC'), Period('2003', 'A-DEC'), Period('2004', 'A-DEC'), Period('2005', 'A-DEC'), Period('2006', 'A-DEC'), Period('2007', 'A-DEC'), Period('2008', 'A-DEC'), Period('2009', 'A-DEC')], dtype=object)

In [27]: all(i.array()==i.values) Out[27]: True

underlying:

In [28]: i._values Out[28]: array([30, 31, 32, 33, 34, 35, 36, 37, 38, 39]) ```

Problem

In pandas, we limit directly calling .values from outside Index, instead accessing Index functions through a smaller API.

But in xarray, I think there are a fair few functions that call .values or implicitly call .__array__() by passing the index into numpy.

As a result, there is a severe slow down when using PeriodIndex. As an example:

```python In [51]: indexes = [pd.PeriodIndex(start=str((1776 + i)), freq='A', periods=300) for i in range(50)] In [53]: das = [xr.DataArray(range(300), coords=[index]) for index in indexes]

In [54]: %timeit xr.concat(das)

1 loop, best of 3: 1.38 s per loop

```

vs DTI:

```python In [55]: indexes_dt = [pd.DatetimeIndex(start=str((1776 + i)), freq='A', periods=300) for i in range(50)] In [56]: das_dt = [xr.DataArray(range(300), coords=[index]) for index in indexes_dt] In [57]: %timeit xr.concat(das_dt)

10 loops, best of 3: 69.2 ms per loop

```

...a 20x slowdown, on fairly short indexes

@shoyer do you have any ideas of how to resolve this? Is it feasible to not pass Indexes directly into numpy? I haven't gone through in enough depth to have a view there, given I was hoping you could cut through the options. Thank you.

ref https://github.com/pandas-dev/pandas/issues/14822 CC @sinhkrs @jreback

This is the repr for a DataArray with 7 coordinates and 5 dimensions. Do we agree this should be shorter?

``` python <xray.DataArray (a: 2, b: 2, c: 5, d: 6771, e: 58)> array([[[[[ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], ..., [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]]],


    [[[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]]]],



   [[[[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]]],


    [[[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

     [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      ..., 
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan],
      [ nan,  nan,  nan, ...,  nan,  nan,  nan]]]]])

Coordinates: ... ```

For an item to be a value in a Dataset, it either needs to be array-like, or pass lib.is_scalar.

I think this is probably too strict - I'd propose anything can be a value, and it only gets treated as an array if it looks like one - thoughts?

Here's the code that checks the values: https://github.com/pydata/xarray/blob/master/xarray/core/variable.py#L52

``` python In [13]: class B(object): pass ....:

In [14]: xr.Dataset({'a':B()})

ValueError Traceback (most recent call last) <ipython-input-14-b2eeb6de19ea> in <module>() ----> 1 xr.Dataset({'a':B()})

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/xarray/core/dataset.py in init(self, data_vars, coords, attrs, compat, **kwargs) 207 coords = set() 208 if data_vars is not None or coords is not None: --> 209 self._set_init_vars_and_dims(data_vars, coords, compat) 210 if attrs is not None: 211 self.attrs = attrs

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/xarray/core/dataset.py in _set_init_vars_and_dims(self, vars, coords, compat) 258 aligned = align_variables(variables) 259 new_variables, new_coord_names = expand_variables(aligned, --> 260 compat=compat) 261 262 new_coord_names.update(coords)

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/xarray/core/merge.py in expand_variables(raw_variables, old_variables, compat) 75 add_variable(dim, coord.variable) 76 var = var.variable ---> 77 add_variable(name, var) 78 79 return new_variables, new_coord_names

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/xarray/core/merge.py in add_variable(name, var) 55 56 def add_variable(name, var): ---> 57 var = _as_dataset_variable(name, var) 58 if name not in variables: 59 variables[name] = var

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/xarray/core/merge.py in _as_dataset_variable(name, var) 9 """ 10 try: ---> 11 var = as_variable(var, key=name) 12 except TypeError: 13 raise TypeError('variables must be given by arrays or a tuple of '

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/xarray/core/variable.py in as_variable(obj, key, strict, copy) 55 obj = Variable(obj.name, obj) 56 elif key is not None: ---> 57 obj = Variable(key, obj) 58 else: 59 raise TypeError('cannot infer Variable dimensions')

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/xarray/core/variable.py in init(self, dims, data, attrs, encoding, fastpath) 211 """ 212 self._data = as_compatible_data(data, fastpath=fastpath) --> 213 self._dims = self._parse_dimensions(dims) 214 self._attrs = None 215 self._encoding = None

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/xarray/core/variable.py in _parse_dimensions(self, dims) 319 raise ValueError('dimensions %s must have the same length as the ' 320 'number of data dimensions, ndim=%s' --> 321 % (dims, self.ndim)) 322 return dims 323

ValueError: dimensions ('a',) must have the same length as the number of data dimensions, ndim=0 ```

Inspired by: https://github.com/pydata/xarray/pull/844. This is a more general case for variable selection.

We could allow a selection of variables with a callable, similar (but not the same) as pandas' implementation:

``` python In [5]: ds=xr.Dataset({'a': (('x', 'y'), np.random.rand(10,2))}) Out[4]: <xarray.Dataset> Dimensions: (x: 10, y: 2) Coordinates: * x (x) int64 0 1 2 3 4 5 6 7 8 9 * y (y) int64 0 1 Data variables: a (x, y) float64 0.5819 0.1214 0.2645 0.9053 0.6968 0.1608 0.3199 ...

In [9]: ds['a'].attrs['clean'] = True

potentially:

In [10]: ds[lambda x: x.attrs['clean']]

... would return ds['a']

```

This would mean functions wouldn't be able to be dataset keys - I don't think that's a big issue, but could arise with callable classes, for example.

Another option would be a .filter / .select method or indexer.