issues

Notes are inline.

• [x] Closes https://github.com/pydata/xarray/issues/8494
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

Done with some help from an LLM — quite good for doing tedious tasks that we otherwise wouldn't want to do — can paste in all the warnings output and get a decent start on rules for exclusions

What is your issue?

I've discussed this a bunch with @dcherian (though I'm not sure he necessarily agrees, I'll let him comment)

Currently xarray uses ddof=0 for std & var. This is: - Rarely what someone actually wants — xarray data is almost always a sample of some underlying distribution, for which ddof=1 is correct - Inconsistent with pandas

OTOH: - It is consistent with numpy - It wouldn't be a painless change — folks who don't read deprecation messages would see values change very slightly

Any thoughts?

Some tests are literally copy & pasted between DataArray & Dataset tests. This change allows them to use a single test. Not everything will be able to use this — sometimes we want to check specifics — but some will — I've change the .cumulative tests to use this fixture.

I was attempting to copy the great work here for numbagg and spotted this! Do we fire twice for each release? Maybe that's fine though?

https://github.com/pydata/xarray/actions/workflows/pypi-release.yaml

Would close https://github.com/pydata/xarray/issues/2570

Because of https://numpy.org/neps/nep-0018-array-function-protocol.html#partial-implementation-of-numpy-s-api, no option is ideal: - Don't do anything — don't implement __array_function__. Any numpy function that's not a ufunc — such as np.clip will materialize the array into memory. - Implement __array_function__ and lose the ability to call any non-ufunc-numpy-func that we don't explicitly configure here. So np.lexsort(da) wouldn't work, for example; and users would have to run np.lexsort(da.values). - Implement __array_function__, and attempt to handle the functions we don't explicitly configure by coercing to numpy arrays. This requires writing code to walk a tree of objects looking for arrays to coerce. It seems to go against the original numpy proposal.

@shoyer is this summary accurate?

Is this reasonable?

• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

This is potentially causing the See also to not render the links? (Does anyone know this better? It doesn't seem easy to build the docs locally...)

This allows resampling with PeriodIndex objects by keeping the group as an index rather than coercing to a DataArray (which coerces any non-native types to objects)

I'm still getting one failure around the name of the IndexVariable still being __resample_dim__ after resample, but wanted to socialize the approach of allowing a name argument to IndexVariable - is this reasonable?

• [x] Closes https://github.com/pydata/xarray/issues/1270
• [x] Tests added
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

This is a very minor nit, so I'm not sure it's worth changing.

What do others think?

(I would have opened an issue but it's just as quick to just do the PR)

In reviewing https://github.com/pydata/xarray/pull/8241, I realize that we actually want check-untyped-defs, which is a bit less strict, but lets us add some more modules on. I did have to add a couple of ignores, think it's a reasonable tradeoff to add big modules like computation on.

Errors with this enabled are actual type errors, not just mypy pedanticness, so would be good to get as much as possible into this list...

I saw that we didn't have this while investigating https://github.com/numba/numba/issues/8615. We should probably wait until that's resolved before merging this (this doesn't solve that issue).

Follow up from the mailing list: - Added some more thoughts on the multi-dimensional comparison. Some of this is opinionated (in concepts not arguments) so I'd appreciate feedback on both the concepts and language. - Removed some of the specific comparison with NDPanel etc, given those are removed - Placeholder for something on the Explicit API (wanted to get this version out, consistent with me leaving less work hanging)

Pandas asserts that the xarray version that's running is above 2022.; locally it was set to 999 which failed this. It only applies when developing locally.

What happened?

Currently if someone supplies a path that doesn't exist, we get quite a long stack trace, without really saying that the path doesn't exist.

What did you expect to happen?

Possibly a FileNotFoundError

Minimal Complete Verifiable Example

Python xr.open_zarr('x.zarr')

Relevant log output

```Python In [1]: xr.open_zarr('x.zarr') <ipython-input-1-8be4b98d9b20>:1: RuntimeWarning: Failed to open Zarr store with consolidated metadata, falling back to try reading non-consolidated metadata. This is typically much slower for opening a dataset. To silence this warning, consider:

1. Consolidating metadata in this existing store with zarr.consolidate_metadata().
2. Explicitly setting consolidated=False, to avoid trying to read consolidate metadata, or
3. Explicitly setting consolidated=True, to raise an error in this case instead of falling back to try reading non-consolidated metadata. xr.open_zarr('x.zarr')

KeyError Traceback (most recent call last) ~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/xarray/backends/zarr.py in open_group(cls, store, mode, synchronizer, group, consolidated, consolidate_on_close, chunk_store, storage_options, append_dim, write_region, safe_chunks, stacklevel) 347 try: --> 348 zarr_group = zarr.open_consolidated(store, **open_kwargs) 349 except KeyError:

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/zarr/convenience.py in open_consolidated(store, metadata_key, mode, **kwargs) 1186 # setup metadata store -> 1187 meta_store = ConsolidatedMetadataStore(store, metadata_key=metadata_key) 1188

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/zarr/storage.py in init(self, store, metadata_key) 2643 # retrieve consolidated metadata -> 2644 meta = json_loads(store[metadata_key]) 2645

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/zarr/storage.py in getitem(self, key) 894 else: --> 895 raise KeyError(key) 896

KeyError: '.zmetadata'

During handling of the above exception, another exception occurred:

GroupNotFoundError Traceback (most recent call last) <ipython-input-1-8be4b98d9b20> in <cell line: 1>() ----> 1 xr.open_zarr('x.zarr')

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/xarray/backends/zarr.py in open_zarr(store, group, synchronizer, chunks, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, consolidated, overwrite_encoded_chunks, chunk_store, storage_options, decode_timedelta, use_cftime, **kwargs) 750 } 751 --> 752 ds = open_dataset( 753 filename_or_obj=store, 754 group=group,

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, backend_kwargs, args, *kwargs) 493 494 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None) --> 495 backend_ds = backend.open_dataset( 496 filename_or_obj, 497 drop_variables=drop_variables,

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/xarray/backends/zarr.py in open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, synchronizer, consolidated, chunk_store, storage_options, stacklevel) 798 799 filename_or_obj = _normalize_path(filename_or_obj) --> 800 store = ZarrStore.open_group( 801 filename_or_obj, 802 group=group,

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/xarray/backends/zarr.py in open_group(cls, store, mode, synchronizer, group, consolidated, consolidate_on_close, chunk_store, storage_options, append_dim, write_region, safe_chunks, stacklevel) 363 stacklevel=stacklevel, 364 ) --> 365 zarr_group = zarr.open_group(store, **open_kwargs) 366 elif consolidated: 367 # TODO: an option to pass the metadata_key keyword

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/zarr/hierarchy.py in open_group(store, mode, cache_attrs, synchronizer, path, chunk_store, storage_options) 1180 if contains_array(store, path=path): 1181 raise ContainsArrayError(path) -> 1182 raise GroupNotFoundError(path) 1183 1184 elif mode == 'w':

GroupNotFoundError: group not found at path '' ```

Anything else we need to know?

No response

Environment

INSTALLED VERSIONS

commit: None python: 3.9.12 (main, Mar 26 2022, 15:44:31) [Clang 13.1.6 (clang-1316.0.21.2)] python-bits: 64 OS: Darwin OS-release: 21.3.0 machine: arm64 processor: arm byteorder: little LC_ALL: en_US.UTF-8 LANG: None LOCALE: ('en_US', 'UTF-8') libhdf5: None libnetcdf: None

xarray: 2022.3.0 pandas: 1.4.1 numpy: 1.22.3 scipy: None netCDF4: None pydap: None h5netcdf: None h5py: None Nio: None zarr: 2.11.1 cftime: None nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: 2021.12.0 distributed: 2021.12.0 matplotlib: None cartopy: None seaborn: None numbagg: None fsspec: 2021.11.1 cupy: None pint: None sparse: None setuptools: 60.9.3 pip: 21.3.1 conda: None pytest: 6.2.5 IPython: 7.32.0 sphinx: None

Reverts pydata/xarray#6543 as discussed.

Merge this in a few days when we know nothing is missing

Currently we get a lot of duplicates:

What is your issue?

From https://github.com/pydata/xarray/pull/6472, and some PRs this weekend:

Is anyone familiar with what's going on with RTD? Did our concurrency limit drop?

Are there alternatives (e.g. running the tests on GHA even if the actual docs get built on RTD?). If we have to pay RTD for a subscription for a bit until we make changes then we could do that (I'm happy to given my recently poor contribution track-record!)

What is your issue?

@pydata/xarray

Because our pipeline takes a while, it can be helpful to have an option to "merge when tests pass" — I've now set that up. So you can click here and it'll do just that.

Someone annoyingly / confusingly, the "required checks" need to be specified manually, in https://github.com/pydata/xarray/settings/branch_protection_rules/2465574 — there's no option for just "all checks".

So if we change the checks — e.g. add Python 3.11 — that list needs to be updated. If we remove a check from the our CI and don't update the list, it won't be possible to merge the PR without clicking the red "Admin Override" box — so we should keep it up to date.

Closes #6497

• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

This auto-fixes problems, so the bar to add it on is low...

• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Following on from https://github.com/pydata/xarray/pull/5690 & https://github.com/pydata/xarray/pull/5694, this type annotates most test files; though probably not most tests — I haven't done the huge files. They require between zero and a dozen fixes, and just ignoring assignment ignores some but not all the errors (I didn't end up using it much).

It also does useful things — fixes a bunch more annotations!

• [x] Passes pre-commit run --all-files

(may require some iteration)

• [x] Passes pre-commit run --all-files

I'm not sure this is the best approach; potentially the comment this app adds is too noisy: https://github.com/marketplace/actions/publish-unit-test-results.

But it would be good to get an idea of the test results from the PR page and to understand where the test suite spends its time

Also adds an example of parameterizing a test in dataset.py

• [x] Passes pre-commit run --all-files

This uses the backend fixture (though not the dataset fixture, like https://github.com/pydata/xarray/pull/5350/files did)

• [x] Passes pre-commit run --all-files

As discussed in other issues — I find this way significantly easier to read, particularly when we're passing tuples of tuples as part of the construction. I did these manually but can write something to do it more broadly if people agree.

I'm about to release this as v0.18.2: https://github.com/pydata/xarray/compare/v0.18.1...max-sixty:release-0.18.2?expand=1 given https://github.com/pydata/xarray/issues/5346

Let me know any thoughts @pydata/xarray , thanks

Is your feature request related to a problem? Please describe.

Currently xarray objects containting datetimes don't operate with pandas' offset objects:

python times = pd.date_range("2000-01-01", freq="6H", periods=10) ds = xr.Dataset( { "foo": (["time", "x", "y"], np.random.randn(10, 5, 3)), "bar": ("time", np.random.randn(10), {"meta": "data"}), "time": times, } ) ds.attrs["dsmeta"] = "dsdata" ds.resample(time="24H").mean("time").time + to_offset("8H")

raises: ```

TypeError Traceback (most recent call last) <ipython-input-29-f9de46fe6c54> in <module> ----> 1 ds.resample(time="24H").mean("time").time + to_offset("8H")

/usr/local/lib/python3.8/site-packages/xarray/core/dataarray.py in func(self, other) 2763 2764 variable = ( -> 2765 f(self.variable, other_variable) 2766 if not reflexive 2767 else f(other_variable, self.variable)

/usr/local/lib/python3.8/site-packages/xarray/core/variable.py in func(self, other) 2128 with np.errstate(all="ignore"): 2129 new_data = ( -> 2130 f(self_data, other_data) 2131 if not reflexive 2132 else f(other_data, self_data)

TypeError: unsupported operand type(s) for +: 'numpy.ndarray' and 'pandas._libs.tslibs.offsets.Hour' ```

This is an issue because pandas resampling has deprecated loffset — from our test suite:

``` xarray/tests/test_dataset.py::TestDataset::test_resample_loffset /Users/maximilian/workspace/xarray/xarray/tests/test_dataset.py:3844: FutureWarning: 'loffset' in .resample() and in Grouper() is deprecated.

df.resample(freq="3s", loffset="8H")

becomes:

from pandas.tseries.frequencies import to_offset df = df.resample(freq="3s").mean() df.index = df.index.to_timestamp() + to_offset("8H")

ds.bar.to_series().resample("24H", loffset="-12H").mean()

```

...and so we'll need to support something like this in order to maintain existing behavior.

Describe the solution you'd like I'm not completely sure; I think probably supporting the operations between xarray objects containing datetime objects and pandas' offset objects.

Numpy warns when computing over an all-NaN slice. We handle that case reasonably and so should handle and discard the warning.

MCVE Code Sample

```python In [1]: import xarray as xr

In [2]: import numpy as np

In [3]: da = xr.DataArray(np.asarray([np.nan]*3))

In [4]: da
Out[4]: <xarray.DataArray (dim_0: 3)> array([nan, nan, nan]) Dimensions without coordinates: dim_0

In [6]: da.mean()
[...]/python3.6/site-packages/xarray/core/nanops.py:142: RuntimeWarning: Mean of empty slice return np.nanmean(a, axis=axis, dtype=dtype) Out[6]: <xarray.DataArray ()> array(nan)

```

Expected Output

No warning

Problem Description

Somewhat discussed in https://github.com/pydata/xarray/issues/1164, and https://github.com/pydata/xarray/issues/1652, but starting a separate issue as it's more important than just noise in the test suite, and not covered by the existing work on comparisons & arithmetic

Output of xr.show_versions()

• [x] Closes #3811
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Is this the right approach? Or should we be using np.errstate machinery?

• [x] closes #3908
• [x] Passes isort . && black . && mypy . && flake8

IIRC assert_identical shows nice pytest error messages.

This was a quick regex but did a quick check that it looked reasonable.

• [x] Passes isort . && black . && mypy . && flake8
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

Inspired by @Illviljan in https://github.com/pydata/xarray/pull/4750

• [x] Closes https://github.com/pydata/xarray/issues/1923 (remove if there is no corresponding issue, which should only be the case for minor changes)
• [x] Tests passed (for all non-documentation changes)

LMK if these are the right ones - I basically added the decorator to anything that was failing. Though not sure we need to be 100% accurate here - worst case we could skip the file - either people are writing backend code and have dask installed, or they're not...

Currently almost none of the docstrings pass running pytest --doctest-modules xarray/core, though mostly for easy reasons.

Should we run these in CI?

I've recently started using docstring tests in another project, and they've work pretty well.

CC @keewis

• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

I had some issues with the version changing — this approach seems more explicit and less likely to cause hard-to-debug issues

• [x] Passes isort . && black . && mypy . && flake8

This PR: - Proposes adding a Release Summary with a PR. This is something that maybe more people read than anything else we send, and yet it's the least reviewed text of anything I write; and I'm making guesses on what's important. - Proposes pasting the release summary into the GH Release page; to the extent people follow xarray by following our releases, it's nice to have something there. - Some touch-ups

Is this a mistake or am I missing something?

http://xray.readthedocs.org/en/stable/generated/xarray.DataArray.count.html?highlight=count#xarray.DataArray.count

skipna : bool, optional If True, skip missing values (as marked by NaN). By default, only skips missing values for float dtypes; other dtypes either do not have a sentinel missing value (int) or skipna=True has not been implemented (object, datetime64 or timedelta64).

I've seen a couple of failures recently on this test. Flaky tests are really annoying and would be great to fix or if impossible, remove it. Does anyone have any ideas what's causing this?

``` __ TestPseudoNetCDFFormat.test_uamiv_format_write __

self = <xarray.tests.test_backends.TestPseudoNetCDFFormat object at 0x7f15352b9d00>

def test_uamiv_format_write(self):
    fmtkw = {"format": "uamiv"}

    expected = open_example_dataset(
        "example.uamiv", engine="pseudonetcdf", backend_kwargs=fmtkw
    )
    with self.roundtrip(
        expected,
        save_kwargs=fmtkw,
        open_kwargs={"backend_kwargs": fmtkw},
        allow_cleanup_failure=True,
    ) as actual:

      assert_identical(expected, actual)

E AssertionError: Left and right Dataset objects are not identical E
E
E
E Differing attributes: E L WTIME: 190117 E R WTIME: 190118

xarray/tests/test_backends.py:3563: AssertionError ```

• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

From the comment:

While we don't generally check the type of every arg, passing multiple dimensions as multiple arguments is common enough, and the consequences hidden enough (strings evaluate as true) to warrant checking here. A future version could make squeeze kwarg only, but would face backward-compat issues.

I don't think these should be needed any longer

I'm less up to speed on this behavior, but IIUC this part of the docstring refers to deprecated behavior—is that correct or am I missing something?

xref https://github.com/pydata/xarray/issues/3266

@crusaderky had a good idea to allow ellipsis (...) as a placeholder for 'other dims' in transpose.

What about using it as a placeholder for xr.ALL_DIMS in groupby etc operations? I find it nicer than custom sentinel values, and I think should be fairly low-confusion—thoughts?

• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Use python 3.6+ idioms; most changes are f-strings

Mostly from pyupgrade --py36-plus **/*.py

• [x] Closes some of #3280
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

This needed updating with the relevant commit post-black change

I also added a line to apply the patch of manual changes we made on top of the black changes. It makes the list of steps a bit burdensome. But I've tested them a couple of times and, where people have lots of code changes, it's still going to be much easier than resolving manually.

I generally wouldn't want to suggest people curl data from the internet (even if we trust the individual). I think it's probably OK in this instance: the address contains a hash of the contents, and it's only being fed into git apply -, not executed. But lmk if that's still a security concern.

I'll update the issue I opened on black with the above issue; we should have done one commit with only the black changes, and then another with any manual changes.

ref https://github.com/pydata/xarray/pull/3090#issuecomment-510323490

Do we want the bot commenting on the PR, at least while the early checks are wrong? People can always click on Details in the Codecov check (e.g. https://codecov.io/gh/pydata/xarray/compare/8f0d9e5c9909c93a90306ed7cb5a80c1c2e1c97d...ab6960f623017afdc99c34bcbb69b402aea3f7d4/diff) to see a full report.

Happy to PR to disable, lmk

https://github.com/pydata/xarray/blob/master/xarray/core/dataset.py#L4085

Should that be called coord or dim? Currently there's a variable that's undefined:

python raise ValueError('Coordinate {} does not exist.'.format(dim)) I would have made a quick fix but not sure the correct name

Currently we use pycodestyle: https://github.com/pydata/xarray/blob/ccd0b047ea8ca89c68ab6cfa942557e676e7d402/.travis.yml#L63

I think we used to use flake8. I can't find / remember the reason we moved to pycodestyle.

master has some non-trivial issues that flake would catch, including a test overwritting another and undefined variables: ``` flake8 xarray --ignore=I,W503,W504,F401,E265,E402

xarray/core/options.py:62:8: F632 use ==/!= to compare str, bytes, and int literals xarray/core/dataset.py:4148:69: F821 undefined name 'dim' xarray/backends/netCDF4_.py:177:12: F632 use ==/!= to compare str, bytes, and int literals xarray/tests/test_dataarray.py:1264:9: F841 local variable 'foo' is assigned to but never used xarray/tests/test_dataarray.py:1270:18: F821 undefined name 'x' xarray/tests/test_dataarray.py:1301:5: F811 redefinition of unused 'test_reindex_fill_value' from line 1262 xarray/tests/test_dataarray.py:1647:16: F632 use ==/!= to compare str, bytes, and int literals xarray/tests/test_dataarray.py:1648:16: F632 use ==/!= to compare str, bytes, and int literals xarray/tests/test_dataset.py:4759:8: F632 use ==/!= to compare str, bytes, and int literals xarray/tests/test_dataset.py:4761:10: F632 use ==/!= to compare str, bytes, and int literals xarray/tests/test_distributed.py:62:9: F811 redefinition of unused 'loop' from line 12 xarray/tests/test_distributed.py:92:9: F811 redefinition of unused 'loop' from line 12 xarray/tests/test_distributed.py:117:49: F811 redefinition of unused 'loop' from line 12 xarray/tests/test_distributed.py:141:53: F811 redefinition of unused 'loop' from line 12 xarray/tests/test_distributed.py:152:51: F811 redefinition of unused 'loop' from line 12 ```

Happy to fix these in a PR. For ensuring these don't crop up again, any objection to flake8?

This: - Updates to the newer GitHub format - Strengthens the language around MCVE. It doesn't say people have to, but makes it less of a suggestion - Only contains 'Bug Report'. Should we have others? I don't immediately see how they'd be different

(I realize because I did this from the GH website, it created from this repo rather than my own fork. That's a mistake.)

• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

message was always going being ignored, and with the newer pytest version raises a warning that an unknown kwarg is supplied

Towards https://github.com/pydata/xarray/issues/2627

```python In [15]: xr.DataArray(np.random.randint(2,size=(100,100)).astype(bool)).shift(dim_0=0) Out[15]: <xarray.DataArray (dim_0: 100, dim_1: 100)> array([[False, True, True, ..., True, True, False], [False, True, False, ..., False, True, True], [False, True, False, ..., False, True, False], ..., [False, True, False, ..., False, True, True], [True, False, True, ..., False, False, False], [False, True, True, ..., True, True, False]], dtype=object) # <-- could be bool Dimensions without coordinates: dim_0, dim_1

```

Problem description

This causes memory bloat

Expected Output

As above with dtype=bool

Output of xr.show_versions()

The shift=0 is mainly theoretical. To avoid casting to object in practical scenarios, we could add a fill_value argument (e.g. fill_value=False) and fill with that rather than NaN

CC @Ivocrnkovic

Still need to adjust the tests

• [x] Closes https://github.com/pydata/xarray/issues/1188
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

@fujiisoup is this a small typo?

• [x] Closes #https://github.com/pydata/xarray/issues/2451
• [x] Tests added
• [x] Tests passed
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

Should we be more defensive around which fill_values can be passed? Currently, if the array and float values have incompatible dtypes, we don't preemtively warn or cast, apart from the case of np.nan, which then uses the default filler

There's an elegant check against the xarray version to decide whether to run a test, so the test 'comes online' at 0.12: https://github.com/pydata/xarray/blob/638b251c622359b665208276a2cb23b0fbc5141b/xarray/tests/test_dataarray.py#L2029

But unfortunately, this seems very unstable in tests, because without a release, LooseVersion can't interpret the strings correctly (e.g. LooseVersion ('0.10.9+29.g33d9391a')) - A lot of the time it raises: https://travis-ci.org/max-sixty/xarray/jobs/437913418#L1036 - Occasionally it runs the test, failing to return that we're prior to 0.12: https://travis-ci.org/max-sixty/xarray/jobs/437914645#L5036

Here's the bug in the python issue tracker: https://bugs.python.org/issue14894

Is that synopsis correct? Should we attempt to take another approach? I'll disable it in my current check so tests can pass, but lmk thoughts.

Do we want to keep isort up to date? I think it's a balance between consistency vs. overhead

If a coordinate is passed as a set, xr raises with a bad error message:

```python

In [12]: xr.Dataset(dict(date=[1,2,3], sec={4}))

MissingDimensionsError Traceback (most recent call last) <ipython-input-12-40ccdd94e21f> in <module>() ----> 1 xr.Dataset(dict(date=[1,2,3], sec={4}))

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/xarray/core/dataset.py in init(self, data_vars, coords, attrs, compat) 360 coords = {} 361 if data_vars is not None or coords is not None: --> 362 self._set_init_vars_and_dims(data_vars, coords, compat) 363 if attrs is not None: 364 self.attrs = attrs

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/xarray/core/dataset.py in _set_init_vars_and_dims(self, data_vars, coords, compat) 378 379 variables, coord_names, dims = merge_data_and_coords( --> 380 data_vars, coords, compat=compat) 381 382 self._variables = variables

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/xarray/core/merge.py in merge_data_and_coords(data, coords, compat, join) 363 objs = [data, coords] 364 explicit_coords = coords.keys() --> 365 return merge_core(objs, compat, join, explicit_coords=explicit_coords) 366 367

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/xarray/core/merge.py in merge_core(objs, compat, join, priority_arg, explicit_coords, indexes) 425 coerced = coerce_pandas_values(objs) 426 aligned = deep_align(coerced, join=join, copy=False, indexes=indexes) --> 427 expanded = expand_variable_dicts(aligned) 428 429 coord_names, noncoord_names = determine_coords(coerced)

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/xarray/core/merge.py in expand_variable_dicts(list_of_variable_dicts) 211 var_dicts.append(coords) 212 --> 213 var = as_variable(var, name=name) 214 sanitized_vars[name] = var 215

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/xarray/core/variable.py in as_variable(obj, name) 103 'cannot set variable %r with %r-dimensional data ' 104 'without explicit dimension names. Pass a tuple of ' --> 105 '(dims, data) instead.' % (name, data.ndim)) 106 obj = Variable(name, obj, fastpath=True) 107 else:

MissingDimensionsError: cannot set variable 'sec' with 0-dimensional data without explicit dimension names. Pass a tuple of (dims, data) instead. But OK if a list:python In [13]: xr.Dataset(dict(date=[1,2,3], sec=[4])) Out[13]: <xarray.Dataset> Dimensions: (date: 3, sec: 1) Coordinates: * date (date) int64 1 2 3 * sec (sec) int64 4 Data variables: empty

```

Problem description

There may be reasons to not allow sets: they're not ordered, so unless you're constructing your data using the coords, you'll get random results

The error message should be better though. And I would vote to handle sets the same as lists

Expected Output

Output of xr.show_versions()

I'd expect a to be a data_vars rather than a coord here:

python In [9]: xr.Dataset(data_vars={'a': [2,3]}, attrs={'name':'hello'}) Out[9]: <xarray.Dataset> Dimensions: (a: 2) Coordinates: * a (a) int64 2 3 Data variables: *empty* Attributes: name: hello

Installing 3.6 environment (from the ci path in xarray): tests pass on master Then installing bottleneck: 3 tests in test_dataarray.py fail on master

I can debug further unless anyone has a view

Potential mitigation for https://github.com/pandas-dev/pandas/issues/14822

Is this intended? It's creating some problems downstream with pandas, but maybe that's a pandas issue?

Note the WRITEABLE : False here:

``` python In [126]: ds=xray.Dataset({'a':xray.DataArray(pd.np.random.rand(5,3))}, coords={'b': xray.DataArray(pd.np.random.rand(5))})

In [127]: ds.to_array('d').b.values.flags Out[127]: C_CONTIGUOUS : True F_CONTIGUOUS : True OWNDATA : False WRITEABLE : False ALIGNED : True UPDATEIFCOPY : False ```

Without the to_array, it's fine:

python In [128]: ds.b.values.flags Out[128]: C_CONTIGUOUS : True F_CONTIGUOUS : True OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False

xref https://github.com/pydata/pandas/issues/11502

I'm not sure on this one. I'm currently having a bunch of issues with this sort of Dataset: (notice the dims are (d, c) and (c, d) for different variables)

python <xray.Dataset> Dimensions: (c: 193, d: 6781) Coordinates: * d (d) object 5218 5219 5220 5221 5222 5223 5224 ... * c (c) object LDS. ... j (c, d) bool False False False False ... Data variables: r (d, c) float64 nan -0.05083 nan ... s (d, c) float64 nan -0.05083 nan ... n (c, d) float64 nan nan nan nan nan ...

In my case, this is particularly painful when passing the result of ds.r.to_pandas() into a function expecting a DataFrame with a certain orientation when that orientation isn't reliable.

Is this a problem generally?

If it is, I could imagine a few solutions - enforce ordering, offer a method on a DataSet to align the dims, offer a kwarg on .to_pandas() to allow specifying the dims-axis mapping, etc

If it is, a Dataset constructor should take a list of DataArrays, and use their names as keys? (and anywhere else you need to provide a dict-like mapping with names)

If it's not, we potentially shouldn't be using it in the internals.

I think it's the first, given it's in the docs (although not throughout the docs).

I'm currently transitioning sequences of pandas Panels over to xray Datasets. Part of our process applies a set of functions to Panels; for example:

python panel = panel.apply(lambda x: x.rank(ascending=False), axis=(1, 2)) df = np.nanpercentile(panel, q=75, axis=2)

One of the benefits of xray is the clarity that comes from named axes. Is there a way of applying a function over named axes? For example:

python result = data_array.apply(np.argsort, axis=('Risk', 'Dates')) result = data_array.apply(np.nanpercentile, axis='Dates')