What is your issue?

I've recently been hit with this when experimenting with xr.dot and xr.corr — xr.dot takes dims, and xr.cov takes dim. Because they each take multiple arrays as positional args, kwargs are more conventional.

Should we standardize on one of these?

What is your issue?

Sorry I haven't been on the calls at all recently (unfortunately the schedule is difficult for me). Maybe this was discussed there? 

PRs now seem to require a separate approval prior to merging. Is there an upside to this? Is there any difference between those who can approve and those who can merge? Otherwise it just seems like more clicking.

PRs also now seem to require merging the latest main prior to merging? I get there's some theoretical value to this, because changes can semantically conflict with each other. But it's extremely rare that this actually happens (can we point to cases?), and it limits the immediacy & throughput of PRs. If the bad outcome does ever happen, we find out quickly when main tests fail and can revert.

(fwiw I wrote a few principles around this down a while ago here; those are much stronger than what I'm suggesting in this issue though)

Is your feature request related to a problem?

Coming back to xarray, and using it based on what I remember from a year ago or so, means I make lots of mistakes. I've also been using it outside of a repl, where error messages are more important, given I can't explore a dataset inline.

Some of the error messages could be much more helpful. Take one example:

xarray.core.merge.MergeError: conflicting values for variable 'date' on objects to be combined. You can skip this check by specifying compat='override'.

The second sentence is nice. But the first could be give us much more information: - Which variables conflict? I'm merging four objects, so would be so helpful to know which are causing the issue. - What is the conflict? Is one a superset and I can join=...? Are they off by 1 or are they completely different types? - Our testing.assert_equal produces pretty nice errors, as a comparison

Having these good is really useful, lets folks stay in the flow while they're working, and it signals that we're a well-built, refined library.

Describe the solution you'd like

I'm not sure the best way to surface the issues — error messages make for less legible contributions than features or bug fixes, and the primary audience for good error messages is often the opposite of those actively developing the library. They're also more difficult to manage as GH issues — there could be scores of marginal issues which would often be out of date.

One thing we do in PRQL is have a file that snapshots error messages test_bad_error_messages.rs, which can then be a nice contribution to change those from bad to good. I'm not sure whether that would work here (python doesn't seem to have a great snapshotter, pytest-regtest is the best I've found; I wrote pytest-accept but requires doctests).

Any other ideas?

Describe alternatives you've considered

No response

Additional context

A couple of specific error-message issues: - https://github.com/pydata/xarray/issues/2078 - https://github.com/pydata/xarray/issues/5290

I did the easy things, but will need help for the final couple on _typed_ops.py

Because we don't pin mypy (should we?), this blocks other PRs if we gate them on mypy passing

• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

Edit: I added a comment outlining a problem with this

I think we should be using a TypeVar(..., bound=...) for our typing, since otherwise subtypes aren't allowed. I don't see a compelling reason to have all of T_DataWithCoords and T_DataArrayOrSet and T_Xarray, though I might be missing something.

So this unifies all T_DataWithCoords, T_DataArrayOrSet and T_Xarray to T_Xarray, and changes that type to be bound on a union of DataArray & Dataset (similar to the existing T_DataArrayOrSet). This covers the bulk of our API — functions which transform either a DataArray or Dataset and return the input type.

This does require some manual casts; I think because when there's a concrete path for both DataArray & Dataset, mypy doesn't unify them back together. It's also possible I'm missing something.

One alternative — a minor change — would be to bound on DataWithCoords, like the existing T_DataWithCoords, rather than the union. A quick comparison showed each required some additional casts / ignores over the other.

What is your issue?

I'm frequently seeing Command killed due to timeout or excessive memory consumption in the doc build.

It's after 1552 seconds, so it not being a round number means it might be the memory?

It follows writing output... [ 90%] generated/xarray.core.rolling.DatasetRolling.max, which I wouldn't have thought as a particularly memory-intensive part of the build?

Here's an example: https://readthedocs.org/projects/xray/builds/21983708/

Any thoughts for what might be going on?

• [x] From https://github.com/pydata/xarray/issues/7164
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

I'm not confident about the CI change; either we can merge with the "this is a trial message" and see how it goes, or split that into a separate PR and discuss.

What happened?

We lose the time coord here — Dimensions without coordinates: time:

```python ds = xr.tutorial.load_dataset("air_temperature") ds.rolling_exp(time=5).mean()

<xarray.Dataset> Dimensions: (lat: 25, time: 2920, lon: 53) Coordinates: * lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0 * lon (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0 Dimensions without coordinates: time Data variables: air (time, lat, lon) float32 241.2 242.5 243.5 ... 296.4 296.1 295.7 ```

(I realize I wrote this, I didn't think this used to happen, but either it always did or I didn't write good enough tests... mea culpa)

What did you expect to happen?

We keep the time coords, like we do for normal rolling:

python In [2]: ds.rolling(time=5).mean() Out[2]: <xarray.Dataset> Dimensions: (lat: 25, lon: 53, time: 2920) Coordinates: * lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0 * lon (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0 * time (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00

Minimal Complete Verifiable Example

Python (as above)

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

• [x] Closes #6391
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

This probably needs a review and another turn, maybe some more tests on multiple objects etc. There are a couple of questions inline.

This replaces https://github.com/pydata/xarray/pull/5787.

Please check out the previews in the most recent comment there

Currently about 40% of the time is taken by installing things, hopefully we can cut that down

• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

IIUC: - This test below would fail mypy because the key of Mapping is invariant (though Any still lets anything through) - I've only seen this when the dict is defined in a different expression from the Dataset constructor - Mypy doesn't type-check our test functions, because the functions don't have any annotations.

If that's correct, we should change all our Mapping to use Any rather than Hashable, and add type annotations to our test functions (even just -> None)

Currently a large share of our tests are in test_dataset.py and test_dataarray.py — each of which are around 7k lines.

There's a case for splitting these up: - Many of the tests are somewhat duplicated between the files (and test_variable.py in some cases) — i.e. we're running the same test over a Dataset & DataArray, but putting them far away from each other in separate files. Should we instead have them split by "function"; e.g. test_rolling.py for all rolling tests? - My editor takes 5-20 seconds to run the linter and save the file. This is a very narrow complaint. - Now that we're all onto pytest, there's no need to have them in the same class.

If we do this, we could start on the margin — new tests around some specific functionality — e.g. join / rolling / reindex / stack (just a few from browsing through) — could go into a new respective test_{}.py file. Rather than some big copy and paste commit.

• [x] Closes #4535 (somewhat)
• [x] Tests added
• [ ] Passes isort . && black . && mypy . && flake8
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

Please make suggestions for any changes!

Follow up to https://github.com/pydata/xarray/pull/2174

Pasting from https://github.com/pydata/xarray/pull/2174#issuecomment-392111566

• [x] stack
• [x] shift
• [x] roll
• [x] set_index
• [x] reorder_levels
• [x] rolling
• [ ] resample (not yet, we still support old behavior for the first positional arguments with a warning)

...potentially rename (I often trip myself up on that)?

• [x] Closes #3789
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

After doing this, I realize it needs to wait until 0.16, assuming 0.15.1 is our next release. If so, this should hang out until then.

We are moving some code over from pandas to Xarray, and one of the biggest missing features is exponential functions, e.g. series.ewm(span=20).mean().

It looks like we can write these as gufuncs without too much trouble in numba. But I also notice that numbagg hasn't changed in a while and that we chose bottleneck for many of the functions in Xarray.

• Is numba a good approach for these?
• As well as our own internal use, could we add numba functions to Xarray, or are there dependency issues?
• Tangentially, I'd be interested why we're using bottleneck rather than numbagg for the existing functions

MCVE Code Sample

```python In [1]: import xarray as xr

In [3]: import numpy as np

In [4]: ds = xr.Dataset( ...: {"foo": (("x", "y"), np.random.rand(4, 3))}, ...: coords={"x": [10, 20, 30, 40], "letters": ("x", list("abba"))}, ...: )

In [5]: ds.groupby('letters')
Out[5]: DatasetGroupBy, grouped over 'letters' 2 groups with labels 'a', 'b'.

In [8]: ds.groupby('letters').sum(...) / ds.groupby('letters').count(...)
Out[8]: <xarray.Dataset> Dimensions: (letters: 2) Coordinates: * letters (letters) object 'a' 'b' Data variables: foo (letters) float64 0.4182 0.4995

In [9]: ds.groupby('letters').pipe(lambda x: x.sum() / x.count())

AttributeError Traceback (most recent call last) <ipython-input-9-c9b142ea051b> in <module> ----> 1 ds.groupby('letters').pipe(lambda x: x.sum() / x.count())

AttributeError: 'DatasetGroupBy' object has no attribute 'pipe'

```

Expected Output

I think we could add groupby.pipe, as a convenience?

Output of xr.show_versions()

A few small tweaks, including a script for getting all recent contributors (feel free to edit if my bash is bad).

• [x] Closes https://github.com/pydata/xarray/issues/1081#issuecomment-540477534
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

HT to @crusaderky for the idea!

(edit: original failure was mistaken) Does anyone know off hand why this is failing?

ResolvePackageNotFound: - pandas=0.19 - python=3.5.0

Worst case we could drop it... https://github.com/pydata/xarray/issues/3293

The only other warning is:

```

xarray/tests/test_dataset.py::TestDataset::test_convert_dataframe_with_many_types_and_multiindex /Users/maximilian/workspace/xarray/xarray/core/dataset.py:3146: FutureWarning: Converting timezone-aware DatetimeArray to timezone-naive ndarray with 'datetime64[ns]' dtype. In the future, this will return an ndarray with 'object' dtype where each element is a 'pandas.Timestamp' with the correct 'tz'. To accept the future behavior, pass 'dtype=object'. To keep the old behavior, pass 'dtype="datetime64[ns]"'. data = np.asarray(series).reshape(shape) /usr/local/lib/python3.7/site-packages/pandas/core/apply.py:286: FutureWarning: Converting timezone-aware DatetimeArray to timezone-naive ndarray with 'datetime64[ns]' dtype. In the future, this will return an ndarray with 'object' dtype where each element is a 'pandas.Timestamp' with the correct 'tz'. To accept the future behavior, pass 'dtype=object'. To keep the old behavior, pass 'dtype="datetime64[ns]"'. results[i] = self.f(v) ```

I'm not sure what we want to do here - potentially we should make a choice between: - the worse behavior of object - accepting the coercing to naive datetimes - supporting pandas extension arrays

master is breaking on both the docs and python3.6 print_versions: https://travis-ci.org/pydata/xarray/jobs/479834129

/home/travis/.travis/job_stages: line 104: 3514 Segmentation fault (core dumped) python xarray/util/print_versions.py The command "python xarray/util/print_versions.py" failed and exited with 139 during .

Has anyone seen this before? I can't replicate locally, but I likely don't have the same dependencies

Based off http://www.rigtorp.se/2011/01/01/rolling-statistics-numpy.html, we wrote up a function that 'tricks' numpy into presenting an array that looks rolling, but without the O^2 memory requirements

Would people be interested in this going into xarray?

It seems to work really well on a few use-cases, but I imagine it's enough trickery that we might not want to support it in xarray. And, to be clear, it's strictly worse where we have rolling algos. But where we don't, you get a rolling apply without the python loops.

```python

def rolling_window_numpy(a, window): """ Make an array appear to be rolling, but using only a view http://www.rigtorp.se/2011/01/01/rolling-statistics-numpy.html """ shape = a.shape[:-1] + (a.shape[-1] - window + 1, window) strides = a.strides + (a.strides[-1],) return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)

def rolling_window(da, span, dim=None, new_dim='dim_0'): """ Adds a rolling dimension to a DataArray using only a view """ original_dims = da.dims da = da.transpose(*tuple(d for d in da.dims if d != dim) + (dim,))

result = apply_ufunc(
    rolling_window_numpy,
    da,
    output_core_dims=((new_dim,),),
    kwargs=(dict(window=span)))

return result.transpose(*(original_dims + (new_dim,)))

tests

import numpy as np import pandas as pd import pytest import xarray as xr

@pytest.fixture def da(dims): return xr.DataArray( np.random.rand(5, 10, 15), dims=(list('abc'))).transpose(*dims)

@pytest.fixture(params=[ list('abc'), list('bac'), list('cab'), ]) def dims(request): return request.param

def test_iterate_imputation_fills_missing(sample_data): sample_data.iloc[2, 2] = pd.np.nan result = iterate_imputation(sample_data) assert result.shape == sample_data.shape assert result.notnull().values.all()

def test_rolling_window(da, dims):

result = rolling_window(da, 3, dim='c', new_dim='x')

assert result.transpose(*list('abcx')).shape == (5, 10, 13, 3)

# should be a view, so doesn't have any larger strides
assert np.max(result.values.strides) == 10 * 15 * 8

def test_rolling_window_values():

da = xr.DataArray(np.arange(12).reshape(2, 6), dims=('item', 'date'))

rolling = rolling_window(da, 3, dim='date', new_dim='rolling_date')

expected = sum([11, 10, 9])
result = rolling.sum('rolling_date').isel(item=1, date=-1)
assert result == expected

```

I was still getting stickler errors for code I hadn't changed. Flake8 should now pass

• [x] Closes #1651
• [x] Tests added / passed
• [ ] Passes git diff upstream/master **/*py | flake8 --diff
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

No docs / docstrings. No backfill.

But otherwise is this a reasonable layout? Do we prefer fillna with options (back / forward / replacement), or ffill, bfill, fillna?

A start for some docs on transitioning from pandas Panel to xarray.

This is some way from the final version - but putting it out there and will iterate.

Is this an inconsistency?

With DatetimeIndex, .item() gets the item, and .data gets a 0-d array:

``` python In [14]: da=xr.DataArray(pd.DataFrame(pd.np.random.rand(10), index=pd.DatetimeIndex(start='2000', periods=10,freq='A')))

In [15]: p=da['dim_0'][0]

In [16]: p.values Out[16]: numpy.datetime64('2000-12-31T00:00:00.000000000')

In [17]: p.item() Out[17]: 978220800000000000L ```

But with a PeriodIndex, values gets the item, and so .item() fails

``` python In [22]: da=xr.DataArray(pd.DataFrame(pd.np.random.rand(10), index=pd.PeriodIndex(start='2000', periods=10)))

In [23]: p=da['dim_0'][0]

In [24]: p.values Out[24]: Period('2000', 'A-DEC')

In [25]: p.item() AttributeError: 'pandas._period.Period' object has no attribute 'item' ```

Not sure if this was a deliberate choice (maybe so integer indexing wasn't disrupted?). This allows objects as dataset keys / labels.

Let me know your thoughts and I'll add a what's new.

(I know I have a few PRs dangling - apologies - will go back and clean them up soon)

Resolves the most pressing issue in #645