issues

What is your issue?

I've discussed this a bunch with @dcherian (though I'm not sure he necessarily agrees, I'll let him comment)

Currently xarray uses ddof=0 for std & var. This is: - Rarely what someone actually wants — xarray data is almost always a sample of some underlying distribution, for which ddof=1 is correct - Inconsistent with pandas

OTOH: - It is consistent with numpy - It wouldn't be a painless change — folks who don't read deprecation messages would see values change very slightly

Any thoughts?

I was attempting to copy the great work here for numbagg and spotted this! Do we fire twice for each release? Maybe that's fine though?

https://github.com/pydata/xarray/actions/workflows/pypi-release.yaml

What happened?

Currently if someone supplies a path that doesn't exist, we get quite a long stack trace, without really saying that the path doesn't exist.

What did you expect to happen?

Possibly a FileNotFoundError

Minimal Complete Verifiable Example

Python xr.open_zarr('x.zarr')

Relevant log output

```Python In [1]: xr.open_zarr('x.zarr') <ipython-input-1-8be4b98d9b20>:1: RuntimeWarning: Failed to open Zarr store with consolidated metadata, falling back to try reading non-consolidated metadata. This is typically much slower for opening a dataset. To silence this warning, consider:

1. Consolidating metadata in this existing store with zarr.consolidate_metadata().
2. Explicitly setting consolidated=False, to avoid trying to read consolidate metadata, or
3. Explicitly setting consolidated=True, to raise an error in this case instead of falling back to try reading non-consolidated metadata. xr.open_zarr('x.zarr')

KeyError Traceback (most recent call last) ~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/xarray/backends/zarr.py in open_group(cls, store, mode, synchronizer, group, consolidated, consolidate_on_close, chunk_store, storage_options, append_dim, write_region, safe_chunks, stacklevel) 347 try: --> 348 zarr_group = zarr.open_consolidated(store, **open_kwargs) 349 except KeyError:

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/zarr/convenience.py in open_consolidated(store, metadata_key, mode, **kwargs) 1186 # setup metadata store -> 1187 meta_store = ConsolidatedMetadataStore(store, metadata_key=metadata_key) 1188

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/zarr/storage.py in init(self, store, metadata_key) 2643 # retrieve consolidated metadata -> 2644 meta = json_loads(store[metadata_key]) 2645

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/zarr/storage.py in getitem(self, key) 894 else: --> 895 raise KeyError(key) 896

KeyError: '.zmetadata'

During handling of the above exception, another exception occurred:

GroupNotFoundError Traceback (most recent call last) <ipython-input-1-8be4b98d9b20> in <cell line: 1>() ----> 1 xr.open_zarr('x.zarr')

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/xarray/backends/zarr.py in open_zarr(store, group, synchronizer, chunks, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, consolidated, overwrite_encoded_chunks, chunk_store, storage_options, decode_timedelta, use_cftime, **kwargs) 750 } 751 --> 752 ds = open_dataset( 753 filename_or_obj=store, 754 group=group,

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, backend_kwargs, args, *kwargs) 493 494 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None) --> 495 backend_ds = backend.open_dataset( 496 filename_or_obj, 497 drop_variables=drop_variables,

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/xarray/backends/zarr.py in open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, synchronizer, consolidated, chunk_store, storage_options, stacklevel) 798 799 filename_or_obj = _normalize_path(filename_or_obj) --> 800 store = ZarrStore.open_group( 801 filename_or_obj, 802 group=group,

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/xarray/backends/zarr.py in open_group(cls, store, mode, synchronizer, group, consolidated, consolidate_on_close, chunk_store, storage_options, append_dim, write_region, safe_chunks, stacklevel) 363 stacklevel=stacklevel, 364 ) --> 365 zarr_group = zarr.open_group(store, **open_kwargs) 366 elif consolidated: 367 # TODO: an option to pass the metadata_key keyword

~/Library/Caches/pypoetry/virtualenvs/-x204KUJE-py3.9/lib/python3.9/site-packages/zarr/hierarchy.py in open_group(store, mode, cache_attrs, synchronizer, path, chunk_store, storage_options) 1180 if contains_array(store, path=path): 1181 raise ContainsArrayError(path) -> 1182 raise GroupNotFoundError(path) 1183 1184 elif mode == 'w':

GroupNotFoundError: group not found at path '' ```

Anything else we need to know?

No response

Environment

INSTALLED VERSIONS

commit: None python: 3.9.12 (main, Mar 26 2022, 15:44:31) [Clang 13.1.6 (clang-1316.0.21.2)] python-bits: 64 OS: Darwin OS-release: 21.3.0 machine: arm64 processor: arm byteorder: little LC_ALL: en_US.UTF-8 LANG: None LOCALE: ('en_US', 'UTF-8') libhdf5: None libnetcdf: None

xarray: 2022.3.0 pandas: 1.4.1 numpy: 1.22.3 scipy: None netCDF4: None pydap: None h5netcdf: None h5py: None Nio: None zarr: 2.11.1 cftime: None nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: 2021.12.0 distributed: 2021.12.0 matplotlib: None cartopy: None seaborn: None numbagg: None fsspec: 2021.11.1 cupy: None pint: None sparse: None setuptools: 60.9.3 pip: 21.3.1 conda: None pytest: 6.2.5 IPython: 7.32.0 sphinx: None

What is your issue?

From https://github.com/pydata/xarray/pull/6472, and some PRs this weekend:

Is anyone familiar with what's going on with RTD? Did our concurrency limit drop?

Are there alternatives (e.g. running the tests on GHA even if the actual docs get built on RTD?). If we have to pay RTD for a subscription for a bit until we make changes then we could do that (I'm happy to given my recently poor contribution track-record!)

What is your issue?

@pydata/xarray

Because our pipeline takes a while, it can be helpful to have an option to "merge when tests pass" — I've now set that up. So you can click here and it'll do just that.

Someone annoyingly / confusingly, the "required checks" need to be specified manually, in https://github.com/pydata/xarray/settings/branch_protection_rules/2465574 — there's no option for just "all checks".

So if we change the checks — e.g. add Python 3.11 — that list needs to be updated. If we remove a check from the our CI and don't update the list, it won't be possible to merge the PR without clicking the red "Admin Override" box — so we should keep it up to date.

I'm about to release this as v0.18.2: https://github.com/pydata/xarray/compare/v0.18.1...max-sixty:release-0.18.2?expand=1 given https://github.com/pydata/xarray/issues/5346

Let me know any thoughts @pydata/xarray , thanks

Is your feature request related to a problem? Please describe.

Currently xarray objects containting datetimes don't operate with pandas' offset objects:

python times = pd.date_range("2000-01-01", freq="6H", periods=10) ds = xr.Dataset( { "foo": (["time", "x", "y"], np.random.randn(10, 5, 3)), "bar": ("time", np.random.randn(10), {"meta": "data"}), "time": times, } ) ds.attrs["dsmeta"] = "dsdata" ds.resample(time="24H").mean("time").time + to_offset("8H")

raises: ```

TypeError Traceback (most recent call last) <ipython-input-29-f9de46fe6c54> in <module> ----> 1 ds.resample(time="24H").mean("time").time + to_offset("8H")

/usr/local/lib/python3.8/site-packages/xarray/core/dataarray.py in func(self, other) 2763 2764 variable = ( -> 2765 f(self.variable, other_variable) 2766 if not reflexive 2767 else f(other_variable, self.variable)

/usr/local/lib/python3.8/site-packages/xarray/core/variable.py in func(self, other) 2128 with np.errstate(all="ignore"): 2129 new_data = ( -> 2130 f(self_data, other_data) 2131 if not reflexive 2132 else f(other_data, self_data)

TypeError: unsupported operand type(s) for +: 'numpy.ndarray' and 'pandas._libs.tslibs.offsets.Hour' ```

This is an issue because pandas resampling has deprecated loffset — from our test suite:

``` xarray/tests/test_dataset.py::TestDataset::test_resample_loffset /Users/maximilian/workspace/xarray/xarray/tests/test_dataset.py:3844: FutureWarning: 'loffset' in .resample() and in Grouper() is deprecated.

df.resample(freq="3s", loffset="8H")

becomes:

from pandas.tseries.frequencies import to_offset df = df.resample(freq="3s").mean() df.index = df.index.to_timestamp() + to_offset("8H")

ds.bar.to_series().resample("24H", loffset="-12H").mean()

```

...and so we'll need to support something like this in order to maintain existing behavior.

Describe the solution you'd like I'm not completely sure; I think probably supporting the operations between xarray objects containing datetime objects and pandas' offset objects.

Numpy warns when computing over an all-NaN slice. We handle that case reasonably and so should handle and discard the warning.

MCVE Code Sample

```python In [1]: import xarray as xr

In [2]: import numpy as np

In [3]: da = xr.DataArray(np.asarray([np.nan]*3))

In [4]: da
Out[4]: <xarray.DataArray (dim_0: 3)> array([nan, nan, nan]) Dimensions without coordinates: dim_0

In [6]: da.mean()
[...]/python3.6/site-packages/xarray/core/nanops.py:142: RuntimeWarning: Mean of empty slice return np.nanmean(a, axis=axis, dtype=dtype) Out[6]: <xarray.DataArray ()> array(nan)

```

Expected Output

No warning

Problem Description

Somewhat discussed in https://github.com/pydata/xarray/issues/1164, and https://github.com/pydata/xarray/issues/1652, but starting a separate issue as it's more important than just noise in the test suite, and not covered by the existing work on comparisons & arithmetic

Output of xr.show_versions()

Currently almost none of the docstrings pass running pytest --doctest-modules xarray/core, though mostly for easy reasons.

Should we run these in CI?

I've recently started using docstring tests in another project, and they've work pretty well.

CC @keewis

Is this a mistake or am I missing something?

http://xray.readthedocs.org/en/stable/generated/xarray.DataArray.count.html?highlight=count#xarray.DataArray.count

skipna : bool, optional If True, skip missing values (as marked by NaN). By default, only skips missing values for float dtypes; other dtypes either do not have a sentinel missing value (int) or skipna=True has not been implemented (object, datetime64 or timedelta64).

I've seen a couple of failures recently on this test. Flaky tests are really annoying and would be great to fix or if impossible, remove it. Does anyone have any ideas what's causing this?

``` __ TestPseudoNetCDFFormat.test_uamiv_format_write __

self = <xarray.tests.test_backends.TestPseudoNetCDFFormat object at 0x7f15352b9d00>

def test_uamiv_format_write(self):
    fmtkw = {"format": "uamiv"}

    expected = open_example_dataset(
        "example.uamiv", engine="pseudonetcdf", backend_kwargs=fmtkw
    )
    with self.roundtrip(
        expected,
        save_kwargs=fmtkw,
        open_kwargs={"backend_kwargs": fmtkw},
        allow_cleanup_failure=True,
    ) as actual:

      assert_identical(expected, actual)

E AssertionError: Left and right Dataset objects are not identical E
E
E
E Differing attributes: E L WTIME: 190117 E R WTIME: 190118

xarray/tests/test_backends.py:3563: AssertionError ```

@crusaderky had a good idea to allow ellipsis (...) as a placeholder for 'other dims' in transpose.

What about using it as a placeholder for xr.ALL_DIMS in groupby etc operations? I find it nicer than custom sentinel values, and I think should be fairly low-confusion—thoughts?

ref https://github.com/pydata/xarray/pull/3090#issuecomment-510323490

Do we want the bot commenting on the PR, at least while the early checks are wrong? People can always click on Details in the Codecov check (e.g. https://codecov.io/gh/pydata/xarray/compare/8f0d9e5c9909c93a90306ed7cb5a80c1c2e1c97d...ab6960f623017afdc99c34bcbb69b402aea3f7d4/diff) to see a full report.

Happy to PR to disable, lmk

https://github.com/pydata/xarray/blob/master/xarray/core/dataset.py#L4085

Should that be called coord or dim? Currently there's a variable that's undefined:

python raise ValueError('Coordinate {} does not exist.'.format(dim)) I would have made a quick fix but not sure the correct name

Currently we use pycodestyle: https://github.com/pydata/xarray/blob/ccd0b047ea8ca89c68ab6cfa942557e676e7d402/.travis.yml#L63

I think we used to use flake8. I can't find / remember the reason we moved to pycodestyle.

master has some non-trivial issues that flake would catch, including a test overwritting another and undefined variables: ``` flake8 xarray --ignore=I,W503,W504,F401,E265,E402

xarray/core/options.py:62:8: F632 use ==/!= to compare str, bytes, and int literals xarray/core/dataset.py:4148:69: F821 undefined name 'dim' xarray/backends/netCDF4_.py:177:12: F632 use ==/!= to compare str, bytes, and int literals xarray/tests/test_dataarray.py:1264:9: F841 local variable 'foo' is assigned to but never used xarray/tests/test_dataarray.py:1270:18: F821 undefined name 'x' xarray/tests/test_dataarray.py:1301:5: F811 redefinition of unused 'test_reindex_fill_value' from line 1262 xarray/tests/test_dataarray.py:1647:16: F632 use ==/!= to compare str, bytes, and int literals xarray/tests/test_dataarray.py:1648:16: F632 use ==/!= to compare str, bytes, and int literals xarray/tests/test_dataset.py:4759:8: F632 use ==/!= to compare str, bytes, and int literals xarray/tests/test_dataset.py:4761:10: F632 use ==/!= to compare str, bytes, and int literals xarray/tests/test_distributed.py:62:9: F811 redefinition of unused 'loop' from line 12 xarray/tests/test_distributed.py:92:9: F811 redefinition of unused 'loop' from line 12 xarray/tests/test_distributed.py:117:49: F811 redefinition of unused 'loop' from line 12 xarray/tests/test_distributed.py:141:53: F811 redefinition of unused 'loop' from line 12 xarray/tests/test_distributed.py:152:51: F811 redefinition of unused 'loop' from line 12 ```

Happy to fix these in a PR. For ensuring these don't crop up again, any objection to flake8?

```python In [15]: xr.DataArray(np.random.randint(2,size=(100,100)).astype(bool)).shift(dim_0=0) Out[15]: <xarray.DataArray (dim_0: 100, dim_1: 100)> array([[False, True, True, ..., True, True, False], [False, True, False, ..., False, True, True], [False, True, False, ..., False, True, False], ..., [False, True, False, ..., False, True, True], [True, False, True, ..., False, False, False], [False, True, True, ..., True, True, False]], dtype=object) # <-- could be bool Dimensions without coordinates: dim_0, dim_1

```

Problem description

This causes memory bloat

Expected Output

As above with dtype=bool

Output of xr.show_versions()

The shift=0 is mainly theoretical. To avoid casting to object in practical scenarios, we could add a fill_value argument (e.g. fill_value=False) and fill with that rather than NaN

CC @Ivocrnkovic

There's an elegant check against the xarray version to decide whether to run a test, so the test 'comes online' at 0.12: https://github.com/pydata/xarray/blob/638b251c622359b665208276a2cb23b0fbc5141b/xarray/tests/test_dataarray.py#L2029

But unfortunately, this seems very unstable in tests, because without a release, LooseVersion can't interpret the strings correctly (e.g. LooseVersion ('0.10.9+29.g33d9391a')) - A lot of the time it raises: https://travis-ci.org/max-sixty/xarray/jobs/437913418#L1036 - Occasionally it runs the test, failing to return that we're prior to 0.12: https://travis-ci.org/max-sixty/xarray/jobs/437914645#L5036

Here's the bug in the python issue tracker: https://bugs.python.org/issue14894

Is that synopsis correct? Should we attempt to take another approach? I'll disable it in my current check so tests can pass, but lmk thoughts.

If a coordinate is passed as a set, xr raises with a bad error message:

```python

In [12]: xr.Dataset(dict(date=[1,2,3], sec={4}))

MissingDimensionsError Traceback (most recent call last) <ipython-input-12-40ccdd94e21f> in <module>() ----> 1 xr.Dataset(dict(date=[1,2,3], sec={4}))

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/xarray/core/dataset.py in init(self, data_vars, coords, attrs, compat) 360 coords = {} 361 if data_vars is not None or coords is not None: --> 362 self._set_init_vars_and_dims(data_vars, coords, compat) 363 if attrs is not None: 364 self.attrs = attrs

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/xarray/core/dataset.py in _set_init_vars_and_dims(self, data_vars, coords, compat) 378 379 variables, coord_names, dims = merge_data_and_coords( --> 380 data_vars, coords, compat=compat) 381 382 self._variables = variables

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/xarray/core/merge.py in merge_data_and_coords(data, coords, compat, join) 363 objs = [data, coords] 364 explicit_coords = coords.keys() --> 365 return merge_core(objs, compat, join, explicit_coords=explicit_coords) 366 367

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/xarray/core/merge.py in merge_core(objs, compat, join, priority_arg, explicit_coords, indexes) 425 coerced = coerce_pandas_values(objs) 426 aligned = deep_align(coerced, join=join, copy=False, indexes=indexes) --> 427 expanded = expand_variable_dicts(aligned) 428 429 coord_names, noncoord_names = determine_coords(coerced)

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/xarray/core/merge.py in expand_variable_dicts(list_of_variable_dicts) 211 var_dicts.append(coords) 212 --> 213 var = as_variable(var, name=name) 214 sanitized_vars[name] = var 215

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/xarray/core/variable.py in as_variable(obj, name) 103 'cannot set variable %r with %r-dimensional data ' 104 'without explicit dimension names. Pass a tuple of ' --> 105 '(dims, data) instead.' % (name, data.ndim)) 106 obj = Variable(name, obj, fastpath=True) 107 else:

MissingDimensionsError: cannot set variable 'sec' with 0-dimensional data without explicit dimension names. Pass a tuple of (dims, data) instead. But OK if a list:python In [13]: xr.Dataset(dict(date=[1,2,3], sec=[4])) Out[13]: <xarray.Dataset> Dimensions: (date: 3, sec: 1) Coordinates: * date (date) int64 1 2 3 * sec (sec) int64 4 Data variables: empty

```

Problem description

There may be reasons to not allow sets: they're not ordered, so unless you're constructing your data using the coords, you'll get random results

The error message should be better though. And I would vote to handle sets the same as lists

Expected Output

Output of xr.show_versions()

I'd expect a to be a data_vars rather than a coord here:

python In [9]: xr.Dataset(data_vars={'a': [2,3]}, attrs={'name':'hello'}) Out[9]: <xarray.Dataset> Dimensions: (a: 2) Coordinates: * a (a) int64 2 3 Data variables: *empty* Attributes: name: hello

Installing 3.6 environment (from the ci path in xarray): tests pass on master Then installing bottleneck: 3 tests in test_dataarray.py fail on master

I can debug further unless anyone has a view

Is this intended? It's creating some problems downstream with pandas, but maybe that's a pandas issue?

Note the WRITEABLE : False here:

``` python In [126]: ds=xray.Dataset({'a':xray.DataArray(pd.np.random.rand(5,3))}, coords={'b': xray.DataArray(pd.np.random.rand(5))})

In [127]: ds.to_array('d').b.values.flags Out[127]: C_CONTIGUOUS : True F_CONTIGUOUS : True OWNDATA : False WRITEABLE : False ALIGNED : True UPDATEIFCOPY : False ```

Without the to_array, it's fine:

python In [128]: ds.b.values.flags Out[128]: C_CONTIGUOUS : True F_CONTIGUOUS : True OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False

xref https://github.com/pydata/pandas/issues/11502

I'm not sure on this one. I'm currently having a bunch of issues with this sort of Dataset: (notice the dims are (d, c) and (c, d) for different variables)

python <xray.Dataset> Dimensions: (c: 193, d: 6781) Coordinates: * d (d) object 5218 5219 5220 5221 5222 5223 5224 ... * c (c) object LDS. ... j (c, d) bool False False False False ... Data variables: r (d, c) float64 nan -0.05083 nan ... s (d, c) float64 nan -0.05083 nan ... n (c, d) float64 nan nan nan nan nan ...

In my case, this is particularly painful when passing the result of ds.r.to_pandas() into a function expecting a DataFrame with a certain orientation when that orientation isn't reliable.

Is this a problem generally?

If it is, I could imagine a few solutions - enforce ordering, offer a method on a DataSet to align the dims, offer a kwarg on .to_pandas() to allow specifying the dims-axis mapping, etc

If it is, a Dataset constructor should take a list of DataArrays, and use their names as keys? (and anywhere else you need to provide a dict-like mapping with names)

If it's not, we potentially shouldn't be using it in the internals.

I think it's the first, given it's in the docs (although not throughout the docs).

I'm currently transitioning sequences of pandas Panels over to xray Datasets. Part of our process applies a set of functions to Panels; for example:

python panel = panel.apply(lambda x: x.rank(ascending=False), axis=(1, 2)) df = np.nanpercentile(panel, q=75, axis=2)

One of the benefits of xray is the clarity that comes from named axes. Is there a way of applying a function over named axes? For example:

python result = data_array.apply(np.argsort, axis=('Risk', 'Dates')) result = data_array.apply(np.nanpercentile, axis='Dates')