What is your issue?

I've recently been hit with this when experimenting with xr.dot and xr.corr — xr.dot takes dims, and xr.cov takes dim. Because they each take multiple arrays as positional args, kwargs are more conventional.

Should we standardize on one of these?

What is your issue?

Sorry I haven't been on the calls at all recently (unfortunately the schedule is difficult for me). Maybe this was discussed there? 

PRs now seem to require a separate approval prior to merging. Is there an upside to this? Is there any difference between those who can approve and those who can merge? Otherwise it just seems like more clicking.

PRs also now seem to require merging the latest main prior to merging? I get there's some theoretical value to this, because changes can semantically conflict with each other. But it's extremely rare that this actually happens (can we point to cases?), and it limits the immediacy & throughput of PRs. If the bad outcome does ever happen, we find out quickly when main tests fail and can revert.

(fwiw I wrote a few principles around this down a while ago here; those are much stronger than what I'm suggesting in this issue though)

Is your feature request related to a problem?

Coming back to xarray, and using it based on what I remember from a year ago or so, means I make lots of mistakes. I've also been using it outside of a repl, where error messages are more important, given I can't explore a dataset inline.

Some of the error messages could be much more helpful. Take one example:

xarray.core.merge.MergeError: conflicting values for variable 'date' on objects to be combined. You can skip this check by specifying compat='override'.

The second sentence is nice. But the first could be give us much more information: - Which variables conflict? I'm merging four objects, so would be so helpful to know which are causing the issue. - What is the conflict? Is one a superset and I can join=...? Are they off by 1 or are they completely different types? - Our testing.assert_equal produces pretty nice errors, as a comparison

Having these good is really useful, lets folks stay in the flow while they're working, and it signals that we're a well-built, refined library.

Describe the solution you'd like

I'm not sure the best way to surface the issues — error messages make for less legible contributions than features or bug fixes, and the primary audience for good error messages is often the opposite of those actively developing the library. They're also more difficult to manage as GH issues — there could be scores of marginal issues which would often be out of date.

One thing we do in PRQL is have a file that snapshots error messages test_bad_error_messages.rs, which can then be a nice contribution to change those from bad to good. I'm not sure whether that would work here (python doesn't seem to have a great snapshotter, pytest-regtest is the best I've found; I wrote pytest-accept but requires doctests).

Any other ideas?

Describe alternatives you've considered

No response

Additional context

A couple of specific error-message issues: - https://github.com/pydata/xarray/issues/2078 - https://github.com/pydata/xarray/issues/5290

What is your issue?

I'm frequently seeing Command killed due to timeout or excessive memory consumption in the doc build.

It's after 1552 seconds, so it not being a round number means it might be the memory?

It follows writing output... [ 90%] generated/xarray.core.rolling.DatasetRolling.max, which I wouldn't have thought as a particularly memory-intensive part of the build?

Here's an example: https://readthedocs.org/projects/xray/builds/21983708/

Any thoughts for what might be going on?

What happened?

We lose the time coord here — Dimensions without coordinates: time:

```python ds = xr.tutorial.load_dataset("air_temperature") ds.rolling_exp(time=5).mean()

<xarray.Dataset> Dimensions: (lat: 25, time: 2920, lon: 53) Coordinates: * lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0 * lon (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0 Dimensions without coordinates: time Data variables: air (time, lat, lon) float32 241.2 242.5 243.5 ... 296.4 296.1 295.7 ```

(I realize I wrote this, I didn't think this used to happen, but either it always did or I didn't write good enough tests... mea culpa)

What did you expect to happen?

We keep the time coords, like we do for normal rolling:

python In [2]: ds.rolling(time=5).mean() Out[2]: <xarray.Dataset> Dimensions: (lat: 25, lon: 53, time: 2920) Coordinates: * lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0 * lon (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0 * time (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00

Minimal Complete Verifiable Example

Python (as above)

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Currently a large share of our tests are in test_dataset.py and test_dataarray.py — each of which are around 7k lines.

There's a case for splitting these up: - Many of the tests are somewhat duplicated between the files (and test_variable.py in some cases) — i.e. we're running the same test over a Dataset & DataArray, but putting them far away from each other in separate files. Should we instead have them split by "function"; e.g. test_rolling.py for all rolling tests? - My editor takes 5-20 seconds to run the linter and save the file. This is a very narrow complaint. - Now that we're all onto pytest, there's no need to have them in the same class.

If we do this, we could start on the margin — new tests around some specific functionality — e.g. join / rolling / reindex / stack (just a few from browsing through) — could go into a new respective test_{}.py file. Rather than some big copy and paste commit.

Follow up to https://github.com/pydata/xarray/pull/2174

Pasting from https://github.com/pydata/xarray/pull/2174#issuecomment-392111566

• [x] stack
• [x] shift
• [x] roll
• [x] set_index
• [x] reorder_levels
• [x] rolling
• [ ] resample (not yet, we still support old behavior for the first positional arguments with a warning)

...potentially rename (I often trip myself up on that)?

We are moving some code over from pandas to Xarray, and one of the biggest missing features is exponential functions, e.g. series.ewm(span=20).mean().

It looks like we can write these as gufuncs without too much trouble in numba. But I also notice that numbagg hasn't changed in a while and that we chose bottleneck for many of the functions in Xarray.

• Is numba a good approach for these?
• As well as our own internal use, could we add numba functions to Xarray, or are there dependency issues?
• Tangentially, I'd be interested why we're using bottleneck rather than numbagg for the existing functions

MCVE Code Sample

```python In [1]: import xarray as xr

In [3]: import numpy as np

In [4]: ds = xr.Dataset( ...: {"foo": (("x", "y"), np.random.rand(4, 3))}, ...: coords={"x": [10, 20, 30, 40], "letters": ("x", list("abba"))}, ...: )

In [5]: ds.groupby('letters')
Out[5]: DatasetGroupBy, grouped over 'letters' 2 groups with labels 'a', 'b'.

In [8]: ds.groupby('letters').sum(...) / ds.groupby('letters').count(...)
Out[8]: <xarray.Dataset> Dimensions: (letters: 2) Coordinates: * letters (letters) object 'a' 'b' Data variables: foo (letters) float64 0.4182 0.4995

In [9]: ds.groupby('letters').pipe(lambda x: x.sum() / x.count())

AttributeError Traceback (most recent call last) <ipython-input-9-c9b142ea051b> in <module> ----> 1 ds.groupby('letters').pipe(lambda x: x.sum() / x.count())

AttributeError: 'DatasetGroupBy' object has no attribute 'pipe'

```

Expected Output

I think we could add groupby.pipe, as a convenience?

Output of xr.show_versions()

(edit: original failure was mistaken) Does anyone know off hand why this is failing?

ResolvePackageNotFound: - pandas=0.19 - python=3.5.0

Worst case we could drop it... https://github.com/pydata/xarray/issues/3293

master is breaking on both the docs and python3.6 print_versions: https://travis-ci.org/pydata/xarray/jobs/479834129

/home/travis/.travis/job_stages: line 104: 3514 Segmentation fault (core dumped) python xarray/util/print_versions.py The command "python xarray/util/print_versions.py" failed and exited with 139 during .

Has anyone seen this before? I can't replicate locally, but I likely don't have the same dependencies

Based off http://www.rigtorp.se/2011/01/01/rolling-statistics-numpy.html, we wrote up a function that 'tricks' numpy into presenting an array that looks rolling, but without the O^2 memory requirements

Would people be interested in this going into xarray?

It seems to work really well on a few use-cases, but I imagine it's enough trickery that we might not want to support it in xarray. And, to be clear, it's strictly worse where we have rolling algos. But where we don't, you get a rolling apply without the python loops.

```python

def rolling_window_numpy(a, window): """ Make an array appear to be rolling, but using only a view http://www.rigtorp.se/2011/01/01/rolling-statistics-numpy.html """ shape = a.shape[:-1] + (a.shape[-1] - window + 1, window) strides = a.strides + (a.strides[-1],) return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)

def rolling_window(da, span, dim=None, new_dim='dim_0'): """ Adds a rolling dimension to a DataArray using only a view """ original_dims = da.dims da = da.transpose(*tuple(d for d in da.dims if d != dim) + (dim,))

result = apply_ufunc(
    rolling_window_numpy,
    da,
    output_core_dims=((new_dim,),),
    kwargs=(dict(window=span)))

return result.transpose(*(original_dims + (new_dim,)))

tests

import numpy as np import pandas as pd import pytest import xarray as xr

@pytest.fixture def da(dims): return xr.DataArray( np.random.rand(5, 10, 15), dims=(list('abc'))).transpose(*dims)

@pytest.fixture(params=[ list('abc'), list('bac'), list('cab'), ]) def dims(request): return request.param

def test_iterate_imputation_fills_missing(sample_data): sample_data.iloc[2, 2] = pd.np.nan result = iterate_imputation(sample_data) assert result.shape == sample_data.shape assert result.notnull().values.all()

def test_rolling_window(da, dims):

result = rolling_window(da, 3, dim='c', new_dim='x')

assert result.transpose(*list('abcx')).shape == (5, 10, 13, 3)

# should be a view, so doesn't have any larger strides
assert np.max(result.values.strides) == 10 * 15 * 8

def test_rolling_window_values():

da = xr.DataArray(np.arange(12).reshape(2, 6), dims=('item', 'date'))

rolling = rolling_window(da, 3, dim='date', new_dim='rolling_date')

expected = sum([11, 10, 9])
result = rolling.sum('rolling_date').isel(item=1, date=-1)
assert result == expected

```

Is this an inconsistency?

With DatetimeIndex, .item() gets the item, and .data gets a 0-d array:

``` python In [14]: da=xr.DataArray(pd.DataFrame(pd.np.random.rand(10), index=pd.DatetimeIndex(start='2000', periods=10,freq='A')))

In [15]: p=da['dim_0'][0]

In [16]: p.values Out[16]: numpy.datetime64('2000-12-31T00:00:00.000000000')

In [17]: p.item() Out[17]: 978220800000000000L ```

But with a PeriodIndex, values gets the item, and so .item() fails

``` python In [22]: da=xr.DataArray(pd.DataFrame(pd.np.random.rand(10), index=pd.PeriodIndex(start='2000', periods=10)))

In [23]: p=da['dim_0'][0]

In [24]: p.values Out[24]: Period('2000', 'A-DEC')

In [25]: p.item() AttributeError: 'pandas._period.Period' object has no attribute 'item' ```