Supress some more internal warnings in the test suite.

• [ ] Adresses points 2 and 3 and closes #8612
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Probably not ready for review yet.

What is your issue?

I looked a bit into the frequency string update & found 3 issues we could improve upon.

1. Apart from "M", pandas also deprecated "Y", and "Q", in favor of "YE" and "QE". (And they are discussing renaming "MS" to "MB"). Should we do the same?

2. Should we translate the new freq strings to the old ones if pandas < 2.2 is installed? Otherwise we get the following situation: python import xarray as xr xr.date_range("1600-02-01", periods=3, freq="M") # deprecation warning xr.date_range("1600-02-01", periods=3, freq="ME") # ValueError: Invalid frequency: ME

3. date_range_like can emit deprecation warnings without a way to mitigate them if pandas < 2.2 is installed. (When a DatetimeIndex) is passed. Could be nice to translate the old freq string to the new one without a warning.

I have played around with 2. and 3. and can open a PR if you are on board.

@spencerkclark @aulemahal

• pandas-dev/pandas#55792
• pandas-dev/pandas#55553
• pandas-dev/pandas#56840

• [x] Closes #7104
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Unstacking non-unique MultiIndex can lead to silent data loss, so we raise an error.

What happened?

As per: https://github.com/pydata/xarray/pull/8246#discussion_r1340357405

Passing a negative step in a slice to select a non-chunked zarr-backed datasets raises an error.

What did you expect to happen?

zarr should allow negative step (probably?)

Minimal Complete Verifiable Example

```Python import xarray as xr

create a zarr dataset

air = xr.tutorial.open_dataset("air_temperature") air.to_zarr("test.zarr")

ds = xr.open_dataset("test.zarr", engine="zarr") ds.air[::-1, ].load()

note that this works if the dataset is backed by dask

ds_dask = xr.open_dataset("test.zarr", engine="zarr", chunks="auto") ds_dask.air[::-1, ].load() ```

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python File ~/code/xarray/xarray/core/parallelcompat.py:93, in guess_chunkmanager(manager) 91 if isinstance(manager, str): 92 if manager not in chunkmanagers: ---> 93 raise ValueError( 94 f"unrecognized chunk manager {manager} - must be one of: {list(chunkmanagers)}" 95 ) 97 return chunkmanagers[manager] 98 elif isinstance(manager, ChunkManagerEntrypoint): 99 # already a valid ChunkManager so just pass through

ValueError: unrecognized chunk manager dask - must be one of: [] ```

Anything else we need to know?

The error comes from https://github.com/zarr-developers/zarr-python/blob/6ec746ef1242dd9fec26b128cc0b3455d28ad6f0/zarr/indexing.py#L174 so it would need an upstream fix first.

cc @dcherian is this what you had in mind?

Environment

Fix a warning from ruff concerning the config:

warning: The top-level linter settings are deprecated in favour of their counterparts in the lint section. Please update the following options in pyproject.toml: - 'extend-safe-fixes' -> 'lint.extend-safe-fixes' - 'per-file-ignores' -> 'lint.per-file-ignores'

• [ ] Closes #xxxx
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

This allows negative freq strings as discussed in https://github.com/pydata/xarray/pull/8627#issuecomment-1905981660 Deciding which tests to update was not easy.

The pandas _generate_range function was moved to https://github.com/pandas-dev/pandas/blob/3c96b8ff6d399fbec8d4d533e8e8618c592bb64b/pandas/core/arrays/datetimes.py#L2725 They no longer rollback the end. I had to remove this as well such that the following are eqivalent:

python xr.date_range("2001", "2000", freq="-1YE", calendar="noleap") pd.date_range("2001", "2000", freq="-1YE")

I am slightly nervous about this but all the tests still pass...

Once again cc @spencerkclark

use ruff.flake8-tidy-imports to enforce absolute imports

• https://github.com/MarcoGorelli/absolufy-imports has been archived (no reason given)
• removes a pre-commit hook which should make it faster locally

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

I think we should use extend-exclude instead of exclude for ruff. We can then also remove ".eggs" as this is in the default.

From https://docs.astral.sh/ruff/settings/#exclude:

Note that you'll typically want to use extend-exclude to modify the excluded paths.

Default value: [".bzr", ".direnv", ".eggs", ".git", ".git-rewrite", ".hg", ".mypy_cache", ".nox", ".pants.d", ".pytype", ".ruff_cache", ".svn", ".tox", ".venv", "__pypackages__", "_build", "buck-out", "build", "dist", "node_modules", "venv"]

(I really dislike how github formats toml files... What would be the correct syntax, then?)

I realized that the return value of infer_freq was not updated. #8627 will try to suppress all warnings in the test suite, so this is just the minimal PR.

Sorry for all the spam @spencerkclark

• [x] Adresses point 1 of #8612
• [x] Fixes one of the failures in #8623
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

This renames the frequency string "Y" (formerly "A") to "YE" to achieve pandas parity. It could be better to wait for the conclusion of pandas-dev/pandas#56840 before doing this (but fixing the related failure in #8623 seemed a good reason as any to do it know).

Let me know what you think @spencerkclark @aulemahal

• [x] Closes #7298
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Otherwise da.indexes and the html repr raise a ValueError. I first had "<undefined>" but I think None is better. cc @spencerkclark @keewis

What happened?

The html repr of a cftime array wants to display the "calendar", which it cannot if it is empty.

What did you expect to happen?

No error.

Minimal Complete Verifiable Example

```Python import numpy as np import xarray as xr

data_obs = np.random.randn(3) time_obs = xr.date_range("2000-01-01", periods=3, freq="YS", calendar="noleap")

obs = xr.DataArray(data_obs, coords={"time": time_obs})

o = obs[:0]

xr.core.formatting_html.array_repr(o) ```

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python ValueError Traceback (most recent call last) Input In [1], in <cell line: 12>() 8 obs = xr.DataArray(data_obs, coords={"time": time_obs}) 10 o = obs[:0] ---> 12 xr.core.formatting_html.array_repr(o)

File ~/code/xarray/xarray/core/formatting_html.py:318, in array_repr(arr) 316 if hasattr(arr, "xindexes"): 317 indexes = _get_indexes_dict(arr.xindexes) --> 318 sections.append(index_section(indexes)) 320 sections.append(attr_section(arr.attrs)) 322 return _obj_repr(arr, header_components, sections)

File ~/code/xarray/xarray/core/formatting_html.py:195, in _mapping_section(mapping, name, details_func, max_items_collapse, expand_option_name, enabled) 188 expanded = _get_boolean_with_default( 189 expand_option_name, n_items < max_items_collapse 190 ) 191 collapsed = not expanded 193 return collapsible_section( 194 name, --> 195 details=details_func(mapping), 196 n_items=n_items, 197 enabled=enabled, 198 collapsed=collapsed, 199 )

File ~/code/xarray/xarray/core/formatting_html.py:155, in summarize_indexes(indexes) 154 def summarize_indexes(indexes): --> 155 indexes_li = "".join( 156 f"

{indexes_li}

File ~/code/xarray/xarray/core/formatting_html.py:156, in <genexpr>(.0) 154 def summarize_indexes(indexes): 155 indexes_li = "".join( --> 156 f"

{indexes_li}

File ~/code/xarray/xarray/core/formatting_html.py:140, in summarize_index(coord_names, index) 138 index_id = f"index-{uuid.uuid4()}" 139 preview = escape(inline_index_repr(index)) --> 140 details = short_index_repr_html(index) 142 data_icon = _icon("icon-database") 144 return ( 145 f"

File ~/code/xarray/xarray/core/formatting_html.py:132, in short_index_repr_html(index) 129 if hasattr(index, "repr_html"): 130 return index.repr_html() --> 132 return f"

{escape(repr(index))}

File ~/code/xarray/xarray/core/indexes.py:547, in PandasIndex.repr(self) 546 def repr(self): --> 547 return f"PandasIndex({repr(self.index)})"

File ~/code/xarray/xarray/coding/cftimeindex.py:353, in CFTimeIndex.repr(self) 345 end_str = format_times( 346 self.values[-REPR_ELLIPSIS_SHOW_ITEMS_FRONT_END:], 347 display_width, 348 offset=offset, 349 first_row_offset=offset, 350 ) 351 datastr = "\n".join([front_str, f"{' '*offset}...", end_str]) --> 353 attrs_str = format_attrs(self) 354 # oneliner only if smaller than display_width 355 full_repr_str = f"{klass_name}([{datastr}], {attrs_str})"

File ~/code/xarray/xarray/coding/cftimeindex.py:272, in format_attrs(index, separator) 267 def format_attrs(index, separator=", "): 268 """Format attributes of CFTimeIndex for repr.""" 269 attrs = { 270 "dtype": f"'{index.dtype}'", 271 "length": f"{len(index)}", --> 272 "calendar": f"'{index.calendar}'", 273 "freq": f"'{index.freq}'" if len(index) >= 3 else None, 274 } 276 attrs_str = [f"{k}={v}" for k, v in attrs.items()] 277 attrs_str = f"{separator}".join(attrs_str)

File ~/code/xarray/xarray/coding/cftimeindex.py:698, in CFTimeIndex.calendar(self) 695 """The calendar used by the datetimes in the index.""" 696 from .times import infer_calendar_name --> 698 return infer_calendar_name(self)

File ~/code/xarray/xarray/coding/times.py:374, in infer_calendar_name(dates) 371 return sample.calendar 373 # Error raise if dtype is neither datetime or "O", if cftime is not importable, and if element of 'O' dtype is not cftime. --> 374 raise ValueError("Array does not contain datetime objects.")

ValueError: Array does not contain datetime objects. ```

Anything else we need to know?

Bisected to 7379923de756a2bcc59044d548f8ab7a68b91d4e use _repr_inline_ for indexes that define it.

Environment

By chance I saw that in #4700 the same code block was added twice. I think this can be removed.

cc @andersy005

• [x] Closes #8367
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Uses max with a default, which work with empty iterators, in contrast to if col_items else 0.

What happened?

da.xindexes or da.indexes raises an error when trying to generate the repr if there are no coords (indexes)

What did you expect to happen?

Displaying an empty Mappable?

Minimal Complete Verifiable Example

Python xr.DataArray([3, 5]).indexes xr.DataArray([3, 5]).xindexes

MVCE confirmation

• [x] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [x] Complete example — the example is self-contained, including all data and the text of any traceback.
• [x] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [x] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [x] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

```Python Out[9]: --------------------------------------------------------------------------- ValueError Traceback (most recent call last) File ~/.conda/envs/xarray_dev/lib/python3.10/site-packages/IPython/core/formatters.py:708, in PlainTextFormatter.call(self, obj) 701 stream = StringIO() 702 printer = pretty.RepresentationPrinter(stream, self.verbose, 703 self.max_width, self.newline, 704 max_seq_length=self.max_seq_length, 705 singleton_pprinters=self.singleton_printers, 706 type_pprinters=self.type_printers, 707 deferred_pprinters=self.deferred_printers) --> 708 printer.pretty(obj) 709 printer.flush() 710 return stream.getvalue()

File ~/.conda/envs/xarray_dev/lib/python3.10/site-packages/IPython/lib/pretty.py:410, in RepresentationPrinter.pretty(self, obj) 407 return meth(obj, self, cycle) 408 if cls is not object \ 409 and callable(cls.dict.get('repr')): --> 410 return _repr_pprint(obj, self, cycle) 412 return _default_pprint(obj, self, cycle) 413 finally:

File ~/.conda/envs/xarray_dev/lib/python3.10/site-packages/IPython/lib/pretty.py:778, in repr_pprint(obj, p, cycle) 776 """A pprint that just redirects to the normal repr function.""" 777 # Find newlines and replace them with p.break() --> 778 output = repr(obj) 779 lines = output.splitlines() 780 with p.group():

File ~/code/xarray/xarray/core/indexes.py:1659, in Indexes.repr(self) 1657 def repr(self): 1658 indexes = formatting._get_indexes_dict(self) -> 1659 return formatting.indexes_repr(indexes)

File ~/code/xarray/xarray/core/formatting.py:474, in indexes_repr(indexes, max_rows) 473 def indexes_repr(indexes, max_rows: int | None = None) -> str: --> 474 col_width = _calculate_col_width(chain.from_iterable(indexes)) 476 return _mapping_repr( 477 indexes, 478 "Indexes", (...) 482 max_rows=max_rows, 483 )

File ~/code/xarray/xarray/core/formatting.py:341, in _calculate_col_width(col_items) 340 def _calculate_col_width(col_items): --> 341 max_name_length = max(len(str(s)) for s in col_items) if col_items else 0 342 col_width = max(max_name_length, 7) + 6 343 return col_width

ValueError: max() arg is an empty sequence ```

Anything else we need to know?

No response

Environment

I really should not open this can of worms but per https://github.com/pydata/xarray/issues/4450#issuecomment-697507489:

I'm always confused about whether ds.groupby(..., keep_attrs=True).mean() or ds.groupby(...).mean(keep_attrs=True) is correct. (similarly for rolling, coarsen etc.)

Also as I try to fix the keep_attr behavior in #4510 it would be good to know where they should go. So I tried to figure out how this is currently handled and found the following:

ds.xxx(keep_attrs=True).yyy() - all fixed

ds.xxx().yyy(keep_attrs=True) - coarsen (fixed in #5227) - groupby - groupby_bin - resample - rolling (adjusted in #4510) - rolling_exp (fixed in #4592) - weighted

So the working consensus seems to be to to ds.xxx().yyy(keep_attrs=True) - any comments on that?

(Edit: looking at this it is only half as bad, "only" coarsen, rolling (#4510), and rolling_exp would need to be fixed.)

Detailed analysis

edit: moved rolling after #4510, moved rolling_exp after #4592 and coarsen after #5227

What happened?

I tried a DataArray with more than one dimension coordinate. Unfortunately xr.align fails, which disallows any arithmetic operation - even when the coords are exactly the same.

What did you expect to happen?

No response

Minimal Complete Verifiable Example

```Python import numpy as np import xarray as xr

data = np.arange(12).reshape(3, 4)

y = [10, 20, 30] s = ["a", "b", "c"]

x = [1, 2, 3, 4]

da = xr.DataArray(data, dims=("y", "x"), coords={"x": x, "y": y, "s": ("y", s)}) da = da.set_xindex("s")

xr.align(da, da.y) # errors da + da # errors da + da.x # errors ```

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [ ] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

```Python

ValueError Traceback (most recent call last) /home/mathause/code/mesmer/devel/prepare_for_surfer.ipynb Cell 28 line 1 12 da = xr.DataArray(data, dims=("y", "x"), coords={"x": x, "y": y, "s": ("y", s)}) 13 da = da.set_xindex("s") ---> 15 xr.align(da, da.y) # errors 17 da + da.x # errors

File ~/.conda/envs/mesmer_dev/lib/python3.9/site-packages/xarray/core/alignment.py:888, in align(join, copy, indexes, exclude, fill_value, *objects) 692 """ 693 Given any number of Dataset and/or DataArray objects, returns new 694 objects with aligned indexes and dimension sizes. ref='~/.conda/envs/mesmer_dev/lib/python3.9/site-packages/xarray/core/alignment.py:0'>0;32m (...) 878 879 """ 880 aligner = Aligner( 881 objects, 882 join=join, ref='~/.conda/envs/mesmer_dev/lib/python3.9/site-packages/xarray/core/alignment.py:0'>0;32m (...) 886 fill_value=fill_value, 887 ) --> 888 aligner.align() 889 return aligner.results

File ~/.conda/envs/mesmer_dev/lib/python3.9/site-packages/xarray/core/alignment.py:573, in Aligner.align(self) 571 self.find_matching_indexes() 572 self.find_matching_unindexed_dims() --> 573 self.assert_no_index_conflict() 574 self.align_indexes() 575 self.assert_unindexed_dim_sizes_equal()

File ~/.conda/envs/mesmer_dev/lib/python3.9/site-packages/xarray/core/alignment.py:318, in Aligner.assert_no_index_conflict(self) 314 if dup: 315 items_msg = ", ".join( 316 f"{k!r} ({v} conflicting indexes)" for k, v in dup.items() 317 ) --> 318 raise ValueError( 319 "cannot re-index or align objects with conflicting indexes found for " 320 f"the following {msg}: {items_msg}\n" 321 "Conflicting indexes may occur when\n" 322 "- they relate to different sets of coordinate and/or dimension names\n" 323 "- they don't have the same type\n" 324 "- they may be used to reindex data along common dimensions" 325 )

ValueError: cannot re-index or align objects with conflicting indexes found for the following dimensions: 'y' (2 conflicting indexes) Conflicting indexes may occur when - they relate to different sets of coordinate and/or dimension names - they don't have the same type - they may be used to reindex data along common dimensions ```

Anything else we need to know?

No response

Environment

What happened?

Running an in-memory groupby operation took much longer than expected. Turning off flox fixed this - but I don't think that's the idea ;-)

What did you expect to happen?

flox to be at least on par with our naive implementation

Minimal Complete Verifiable Example

```Python import numpy as np import xarray as xr

arr = np.random.randn(10, 10, 36530) time = xr.date_range("2000", periods=30365, calendar="noleap") da = xr.DataArray(arr, dims=("y", "x", "time"), coords={"time": time})

using max

print("max:") xr.set_options(use_flox=True) %timeit da.groupby("time.year").max("time") %timeit da.groupby("time.year").max("time", engine="flox")

xr.set_options(use_flox=False) %timeit da.groupby("time.year").max("time")

as reference

%timeit [da.sel(time=str(year)).max("time") for year in range(2000, 2030)]

using mean

print("mean:") xr.set_options(use_flox=True) %timeit da.groupby("time.year").mean("time") %timeit da.groupby("time.year").mean("time", engine="flox")

xr.set_options(use_flox=False) %timeit da.groupby("time.year").mean("time")

as reference

%timeit [da.sel(time=str(year)).mean("time") for year in range(2000, 2030)] ```

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python max: 158 ms ± 4.41 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) 28.1 ms ± 318 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) 11.5 ms ± 52.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

mean: 95.6 ms ± 10.8 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) 34.8 ms ± 2.88 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) 15.2 ms ± 232 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) ```

Anything else we need to know?

No response

Environment

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

This makes many arguments keyword-only, except for dim to avoid da.weighted(...).mean("lat", "lon") (i.e. da.weighted(...).mean(dim="lat", skipna="lon")) which silently does the wrong thing. I am sure I forgot some and for some I was unsure so I left them as is.

Question: do we want an deprecation cycle? Currently it just errors for da.weighted(...).mean("dim", True). Might be nice to do it, however, @dcherian if I am not mistaken you did this without a deprecation in #5950, e.g. for da.mean etc.?

python import numpy as np import xarray as xr air = xr.tutorial.open_dataset("air_temperature") wgt = np.cos(np.deg2rad(air.lat)) air.weighted(wgt).mean("lat", "lon")

• [ ] Towards #8239
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

This partly updates the pytest config as suggested in #8239 and un-xfails some tests (or xfails the tests more precisely).

See https://github.com/pydata/xarray/issues/8239#issuecomment-1739363809 for why we cannot exactly follow the suggestions given in #8239

When I manually specify the units attribute for time, and then save the Dataset to_netcdf the string is truncated. See exaple

import pandas as pd
import xarray as xr

time = pd.date_range('2000-01-01', '2000-01-31', freq='6h')
ds = xr.Dataset(coords=dict(time=time))

units = 'days since 1975-01-01 00:00:00'
calendar = 'gregorian'
encoding=dict(time=dict(units=units, calendar=calendar))

ds.to_netcdf('test.nc', format='NETCDF4_CLASSIC', encoding=encoding)

! ncdump -h test.nc
# time:units = "days since 1975-01-01" ;

Some programs seem to require the hours to be present to interpret the time properly (e.g. panoply). When specifying the hour, a 'T' is added.

units = 'days since 1975-01-01 01:00:00'

! ncdump -h test.nc
# time:units = "days since 1975-01-01T01:00:00" ;

When xarray defines the time.units it works fine.

ds = xr.Dataset(coords=dict(time=time))
ds.to_netcdf('test.nc', format='NETCDF4_CLASSIC',)

! ncdump -h test.nc
# time:units = "hours since 2000-01-01 00:00:00" ;

xarray version 0.9.6

• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

I found the error message in ds.to_stacked_array confusing, so I tried to make it clearer. Also renames some if the internal symbols (so should have no user facing change).

Suppress the pkg_resources.namespace_package DeprecationError to make the doctest pass again (similar to #7322). This is reported upstream: pydap/pydap#277 and matplotlib/matplotlib#25244

• [x] Closes #6092
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

One change is that it is no longer possible to compare to a string, i.e. version.parse(xr.__version__) < "0.20.0" errors.

As mentioned in #6092 there are 3 options - if there is a preference I am happy to update this PR.

```python from distutils.version import LooseVersion from packaging import version

LooseVersion(xr.version) version.parse(xr.version) version.Version(xr.version)

currently:

if LooseVersion(mod.version) < LooseVersion(minversion): pass

options:

if version.parse(mod.version) < version.parse(minversion): pass

if version.Version(mod.version) < version.Version(minversion): pass

if Version(mod.version) < Version(minversion): pass ```

• [x] Closes #7322

This PR renames the da and ds fixtures (to da_fixture and ds_fixture) and moves them to conftest.py. This allows to remove the flake8 error suppression for the tests and seems more how the fixtures are intended to be used (from the pytest side). I think the name changes makes it a bit more obvious what happens but moving them to may make it a bit less obvious (if you don't know where to look).

Removing the flake8 error ignores also unearthed some unused imports:

https://github.com/pydata/xarray/blob/787a96c15161c9025182291b672b3d3c5548a6c7/setup.cfg#L155-L156

(What I actually wanted to do is move the tests for rolling to it's own file - but I think it makes sense to do this first.)

I forgot to remove some debug print statements in #6910 - thanks for noting @shoyer & @dcherian

flake8 v6 now errors on inline comments in the config file. I don't like it but oh well...

We are currently not testing with the newest version of netCDF4 because it is incompatible with pynio (the newest version is 1.5.4, we are at 1.5.3). This is unlikely to be fixed, see conda-forge/pynio-feedstock#90.

Therefore we need to think how to setup the tests so we use the newest version of netCDF4. Maybe just remove it from py38.yml?

And long term what to do with the pynio backend? Deprecate? Move to an external repo?

What happened?

xr.core.dataset.DataVariables assumes that everything that is in ds._dataset._variables and not in self._dataset._coord_names is a "data variable". However, since the index refactor we can end up with more _coord_names than _variables which breaks a number of stuff (e.g. the repr).

What did you expect to happen?

Well it seems this assumption is now wrong.

Minimal Complete Verifiable Example

Python ds = xr.Dataset(coords={"a": ("x", [1, 2, 3]), "b": ("x", ['a', 'b', 'c'])}) ds.set_index(z=['a', 'b']).reset_index("z", drop=True)

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

Python ValueError: __len__() should return >= 0

Anything else we need to know?

The error comes from here

https://github.com/pydata/xarray/blob/63ba862d03c8d0cd8b44d2071bc360e9fed4519d/xarray/core/dataset.py#L368

Bisected to #5692 - which probably does not help too much.

Environment

In #4864 I added a a decorator for the xarray.testing.assert_* functions to ensure warnings that were to errors (pytest.mark.filterwarnings("error")) do not error in assert_* (see https://github.com/pydata/xarray/pull/4760#issuecomment-774101639). As a solution I added

https://github.com/pydata/xarray/blob/5470d933452d88deb17cc9294a164c4a03f55dec/xarray/testing.py#L32

However, this is sub-optimal because this now removes all ignore filters! As dask stuff only gets evaluated in assert_* filters like warnings.filterwarnings("ignore", "Mean of empty slice") don't work for dask arrays!

I thought of setting

python warnings.simplefilter("ignore")

but this could suppress warnings we want to keep.

So now I remove all "error" warning filters and keep the rest. Note that the original filters get restored after with warnings.catch_warnings():. ().

I am not sure I expressed myself very clearly... let me know and I can try again. @keewis you had a look at #4864 maybe you can review this PR as well?

I tried to use the _deprecate_positional_args decorator from #6934 & it turns out that it still had some errors - passing on the arguments did not work properly in certain cases... I now added tests for this as well (which I should have done in the first place...).

5844 excluded pydap from our tests - but the new version has been released in the meantime (on conda not on pypi, though, pydap/pydap#268) - so let's see if this still works.

• [x] Closes #6960
• [ ] Tests added - tested manually

Get rid of some warnings in the tests.

• [x] Closes #6932

@zmoon - obviously it would be nice if we had a linter for this but this is for another time.

• [x] Supersedes #6403, see also #5531
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Adds a helper function to deprecate positional arguments. IMHO this offers a good trade-off between magic and complexity. (As mentioned this was adapted from scikit-learn).

edit: I suggest to actually deprecate positional arguments in another PR.

fix some typos (using codespell). Called using:

bash codespell --skip=".git,.mypy_cache,*.tex" --ignore-words-list coo,nd,inferrable,hist,ND,splitted,soler,slowy,ba,ser,nin,te,fo -w -i 3

This started as a question and I add it as reference. Maybe you have a comment.

There are several ways to calculate time series of seasonal data (starting from monthly or daily data):

```

load libraries

import pandas as pd import matplotlib.pyplot import numpy as np import xarray as xr

Create Example Dataset

time = pd.date_range('2000.01.01', '2010.12.31', freq='M') data = np.random.rand(*time.shape) ds = xr.DataArray(data, coords=dict(time=time))

(1) using resample

ds_res = ds.resample('Q-FEB', 'time') ds_res = ds_res.sel(time=ds_res['time.month'] == 2) ds_res = ds_res.groupby('time.year').mean('time')

(2) this is wrong

ds_season = ds.where(ds['time.season'] == 'DJF').groupby('time.year').mean('time')

(3) using where and rolling

mask other months with nan

ds_DJF = ds.where(ds['time.season'] == 'DJF')

rolling mean -> only Jan is not nan

however, we loose Jan/ Feb in the first year and Dec in the last

ds_DJF = ds_DJF.rolling(min_periods=3, center=True, time=3).mean()

make annual mean

ds_DJF = ds_DJF.groupby('time.year').mean('time')

ds_res.plot(marker='*') ds_season.plot() ds_DJF.plot()

plt.show() ```

(1) The first is to use resample with 'Q-FEB' as argument. This works fine. It does include Jan/ Feb in the first year, and Dec in the last year + 1. If this makes sense can be debated. One case where this does not work is when you have, say, two regions in your data set, for one you want to calculate DJF and for the other you want NovDecJan.

(2) Using 'time.season' is wrong as it combines Jan, Feb and Dec from the same year.

(3) The third uses where and rolling and you lose 'incomplete' seasons. If you replace ds.where(ds['time.season'] == 'DJF') with ds.groupby('time.month').where(summer_months), where summer_months is a boolean array it works also for non-standard 'summers' (or seasons) across the globe.

• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

skipna=None did not skip missing values for quantile, inconsistent with other methods. Discovered while testing #6059.

• [x] Closes #6136

Convert the code block in xr.DataArray.from_dict and xr.Dataset.from_dict to doctest/ examples.

pytest v7.0.0 no longer want's us to use pytest.warns(None) to test for no warning, so we can use warnings.catch_warnings(record=True) instead.

numpy/numpy#20327 introduces some changes to np.quantile (and related) for the upcoming numpy release (v1.22.0). It renames the interpolation keyword to method and offers some new interpolation methods. This PR does two things

1. it restores compatibility with numpy 1.22

2. it renames the interpolation keyword to method in xarray - this change is not strictly necessary but I thought better to be consistent with numpy

3. [x] Tests added

4. [x] Passes pre-commit run --all-files
5. [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

(Side note in dask.array.percentile the method keyword is used differently from the interpolation keyword (docs). However, xarray does not use the dask function.)

TODO: need to import ArrayLike from npcompat.

• [x] Closes #6186

pydata/bottleneck#378 was merged - so this should work again.

bottleneck cannot be built with python 3.10. See https://github.com/pydata/xarray/actions/runs/1731371015

Fixes or suppresses a number of warnings that turn up in our upstream CI.

pd.Index.is_monotonic is an alias for pd.Index.is_monotonic_increasing and does not stand for pd.Index.is_monotonic_increasing or pd.Index.is_monotonic_decreasing.

Reverts pydata/xarray#6200

And another tiny one... The somewhat ambiguous date string triggers a warning in pandas which makes our doc build fail.

• [x] Closes #5721
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

Fixed as per @shoyer & @spencerkclark suggestion from https://github.com/pydata/xarray/issues/5721#issuecomment-903095007

Now that pandas 1.4 is out it would be good to get this fixed (there are about 5000 warnings in our tests, mostly because of interp, though). Also leads to a warning in our docs which breaks them (although that can also be fixed with an :okwarning: directive).

pandas deprecates the method keyword in Index.get_loc, see pandas-dev/pandas#42269. Therefore we end up with about 5000 warnings in our upstream tests:

FutureWarning: Passing method to Index.get_loc is deprecated and will raise in a future version. Use index.get_indexer([item], method=...) instead

We should fix this before pandas releases because the warning will not be silent (FutureWarning) or ask pandas to give us more time and use a DeprecationWarning at the moment.

@benbovy @spencerkclark

Fixes the doctests for the newest version of cftime. @spencerkclark

This of course means that the doctests will fail for environments with older versions of cftime present. I don't think there is anything we can do.

Thanks for pytest-accept b.t.w @max-sixty

5740 changed PendingDeprecationWarning to FutureWarning - suppress the warnings again in the test suite.

https://github.com/pydata/xarray/blob/36f05d70c864ee7c61603c8a43ba721bf7f434b3/xarray/ufuncs.py#L47-L49

• [x] Closes #6016

cftime 1.5.2 renames "gregorian" to "standard". AFAIK this only changes the repr of cftime indices and does not seem to influence the creation of cftime indices.

In #6096 I realized that DuckArrayModule("dask") is called a lot in our tests - 145'835 times. Most of those are from is_dask_collection (is_duck_dask_array) This change avoids that the instance needs to be built every time.

```python import xarray as xr

%timeit xr.core.pycompat.DuckArrayModule("dask").available %timeit xr.core.pycompat.dsk.available ```

18.9 µs ± 97.7 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) 77.1 ns ± 1.22 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

Which leads to an incredible speed up of our tests of about 2.7 s :grin: ((18.9 - 0.0771) * 145835 / 1e6).

• [x] Closes #4601
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

It's not pretty as I had to define a number of empty methods... I think this should wait for 0.17

• [x] Closes #4727
• [ ] Tests added
• [ ] Passes isort . && black . && mypy . && flake8
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

This adds a dtype check for equal, identical, broadcast_equal, and the xr.testing.assert_* functions. It is far from complete: tests and documentation are still missing, but I wanted to get it online for feedback.

When I set check_dtype=True there are around 600 failures. Fixing that is for another PR. #4759 should help a bit.

• [ ] I added the checks to lazy_array_equiv, however, sometimes dask can get the dtype wrong before the compute (see below). Do you think I need to put it in the non-lazy part?

```python import numpy as np import xarray as xr

da = xr.DataArray(np.array([0, np.nan], dtype=object)).chunk()

da.prod().dtype # -> dtype('O') da.prod().compute().dtype # -> dtype('int64')

```

• [ ] check_dtype is still missing from assert_duckarray_allclose & assert_duckarray_equal - do you think there are required?

• [ ] The dtypes of array elements are not tested (see below). I don't think I'll implement that here.

```python da0 = xr.DataArray(np.array([0], dtype=object)) da1 = xr.DataArray(np.array([0.], dtype=object))

xr.testting.assert_equal(da0, da1, check_dtype=True) ```

• dask 2022.01 is out, so we can remove the pin

I hunted down some version checks that should no longer be necessary as we have moved beyond the minimum versions.

• [x] see #6186

I think it would be good to re-enable the upstream CI, even if this means we have to stick to py3.9 for the moment. I just subscribed to pydata/bottleneck#378, so I should see when we can switch to 3.10.

typing fixes for mypy 0.931 and numpy 1.22. Also tested with numpy 1.20 which probably many still have installed.

```python import xarray as xr import numpy as np

ds = xr.DataArray([np.NaN, np.NaN])

ds.mean() ds.sum()

```

Problem description

ds.mean() returns NaN, ds.sum() returns 0. This comes from numpy (cp np.nanmean vs. np.nansum), so it might have to be discussed upstream, but I wanted to ask the xarray community first on their opinion. This is also relevant for #422 (what happens if the all weights are NaN or sum up to 0).

Expected Output

I would expect both to return np.nan.

Output of xr.show_versions()

Our CI fails with some pytest-xdist error. Let's see if we get a clearer picture when disabling parallel tests. (Maybe some interaction between dask and pytest-xdist?).

• [x] Closes #6101

Reverts pydata/xarray#6077 (after dask has been pinned in #6111)

I disabled pytest-xdist because a dask issue renders our CI unusable. As soon as dask releases a new version we should revert #6077 again.

h5netcdf no longer warns for invalid netCDF (unless passing save_kwargs = {"invalid_netcdf": True}. We need to adapt our tests.

@kmuehlbauer

edit: I added h5netcdf to the upstream tests - I can also revert this change if you prefer.

• [x] Closes #5872
• [x] Passes pre-commit run --all-files

pandas will disallow skipna=None (pandas-dev/pandas#44178) - this fixes a test which relies on this. I don't think we have any user facing use of this. AFAIK we don't use pandas for reductions anywhere)

Fixes one part of #5872

pseudoNETCDF adds two attrs to ict files, which breaks the following two tests:

Test 1: https://github.com/pydata/xarray/blob/07de257c5884df49335496ee6347fb633a7c302c/xarray/tests/test_backends.py#L3944 Test 2:

https://github.com/pydata/xarray/blob/07de257c5884df49335496ee6347fb633a7c302c/xarray/tests/test_backends.py#L4030

I reproduced the test file so that the tests pass again. To reproduce the file I used the following bit of code:

```python import xarray as xr from xarray.tests import test_backends

fN = "xarray/tests/data/example.ict" fmtkw = {"format": "ffi1001"}

ds = xr.open_dataset(fN, engine="pseudonetcdf", backend_kwargs={"format": "ffi1001"})

c = test_backends.TestPseudoNetCDFFormat() c.save(ds, fN, **fmtkw) ```

The save method is here:

https://github.com/pydata/xarray/blob/07de257c5884df49335496ee6347fb633a7c302c/xarray/tests/test_backends.py#L4124

@barronh I would appreciate your review here - I am not sure if this is the right approach.

• [x] Part of #4513
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

This PR warns when passing keep_attrs to resample(..., keep_attrs=True) and rolling_exp(..., keep_attrs=True) as they have no effect (rightfully). Also removes keep_attrs from the docstring of resample.

• [x] Part of #4513 (maybe the last one - need to double check)
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

As per https://github.com/pydata/xarray/issues/3891#issuecomment-612522628 I also changed the default to True.

Another enormous PR from my side ;) unpin pip again. numpy probably fixed the issue re the name of the nightly build. But I also need to doublecheck if scipy is ok.

• [x] Closes #4858
• [x] Passes pre-commit run --all-files

iris moved its documentation to https://scitools-iris.readthedocs.io/en/stable/

There is a compatibility issue with h5py v3. Pin h5py to version 2 for the moment. I can open an issue shortly.

• [x] Closes #4521
• [x] Passes isort . && black . && mypy . && flake8

That was fast - eccodes already fixed the issue.

and put the comment on the same line

• [x] Closes #3756
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [x] Closes #2019
• [ ] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API
• [x] reverts #2021

Let's see if this passes....

• [x] Closes #3634
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

np.nanquantile was added in numpy 1.15.0, the current minimum requirement for numpy is 1.14.0, therefore we have to test this ourselves.

• [x] Closes #5517
• [x] Passes pre-commit run --all-files

The minimal typing for numpy 1.21. As always I am by no means a typing specialist.

• do not change user-specified units
• always format infered units as 'YYYY-mmmm-ddTHH:MM:SS'

This is my naïve approach.

• [ ] Closes #1449
• [ ] Tests added / passed
• [ ] Passes git diff upstream/master | flake8 --diff
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

Silences a number of warnings that accumulated in our test suite (c.f. #3266). The changes are mostly unrelated but small.

matplotlib warns: Calling gca() with keyword arguments was deprecated in Matplotlib 3.4. Starting two minor releases later, gca() will take no keyword arguments. The gca() function should only be used to get the current axes, or if no axes exist, create new axes with default keyword arguments. To create a new axes with non-default arguments, use plt.axes() or plt.subplot().

This only uses plt.gca() if there are active axes, else it calls plt.axes(**kwargs). Note that this can silently ignore some arguments. However, that this is already the case.

• [x] Closes #5444

This was changed in dask/dask#7669. Looks like they did not deprecate this behavior (i.e. passing both meta and output_dtypes). I'd suggest to follow dask's example here and not add a deprecation cycle. Thoughts?

What happened:

I could not open the test dataset on master. It's a bit strange that this is not picked up by the tests, so probably something to do with the environment I have (I just updated all packages).

@alexamici @aurghs does that tell you anything?

I can also try to figure it out.

Minimal Complete Verifiable Example:

calling open_dataset with "" is enough to trigger the error:

python import xarray as xr air = xr.open_dataset("")

Anything else we need to know?:

And the traceback:

Environment:

and my conda list:

edit: added the traceback

• [x] Closes #4693

• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Adds error codes to all # type: ignore comments. mypy should raise if a different type of error is encountered. Also enable showing the error code for typing errors. See: mypy: displaying-error-codes. Also remove some # type: ignore that are no longer necessary.

• [x] Part of #5077
• [x] Tests added
• [x] Passes pre-commit run --all-files

• [x] Closes #2007
• [ ] Tests added (for all bug fixes or enhancements)
• [ ] Tests passed (for all non-documentation changes)
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

Ok, this was easier to do than initially thought, we can use np.pad(a, pads, mode='wrap') in nputils.rolling_window. However, I'm not sure if that is enough already<sup>*</sup>.

I added an initial test, but could use some pointers where else you want this to be tested.

Questions: * is fill_value='periodic' a good api? * should the fill_value keyvalue be ported to rolling? * should this also be mentioned in the docs for rolling (I only learned about rolling.construct yesterday)

<sup>*</sup>rolling is present incore/dataset.py, core/dataarray.py, core/variable.py, core/rolling.py, core/dask_array_ops.py, core/nputils.py, core/ops.py, core/common.py, core/missing.py, and core/duck_array_ops.py that can be a bit daunting...

• [x] Closes #4970

The clean thing here would be to update min_deps_check.py so it works properly for this case. I am not sure it's really worth it.

We currently pin netCDF4 to version 1.5. However, I think netCDF4 does not really follow semantic versioning, e.g. python 2 support was dropped in version 1.5.6. So they may actually be doing something like 1.major.minor[.patch] - I asked about their versioning scheme in Unidata/netcdf4-python#1090.

So I wonder if we would need to pin netCDF to version to version 1.5.4.

• xref #4945

Try to limit the time of "CI Upstream" to avoid a silent failure.

The last 5 days our Upstream CI failed silently with a timeout after 6h:

https://github.com/pydata/xarray/actions/workflows/upstream-dev-ci.yaml?query=branch%3Amaster+event%3Aschedule

This was probably caused by #4934. As mentioned in dask/dask#4934 this is probably dask/dask#6738 which was merged 5 days ago.

numpy v1.20.0 introduced type hints which leads to some mypy errors in xarray. This is the minimum set of changes to make mypy happy again. I tried to avoid #type: ignore and Any but in some instances I think it was not worth it.

I am sure there is much more fun to be had with numpy typing ;-)

• [x] Closes #4881
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

I only added one run for py3.8 latest. To be entirely sure we could also check the typing py37-min-all-deps but it feels like overkill...

Ok, looks good - the failure is expected - see #4878.

We currently run mypy in the pre-commit hooks CI. However, this is done in an environment where no dependencies are installed. Therefore we missed the errors that pop up when running mypy with numpy 1.20 installed. (Please correct my if I misunderstood this). Should we add a new step to our CI and run mypy?

I think we should at least add this to ubuntu-latest py3.8. For more complete checks we could also go for ubuntu-latest py37-min-all-deps and upstream-dev.

Small doc fix, see http://xarray.pydata.org/en/stable/io.html#formats-supported-by-pynio (the ..note:: did not get picked up)

h5py version 3.1 broke our tests. I pinned it to version 2.10 in #4569. We should therefore

• fix the issues
• unpin h5py again

The failures could be related to a change how strings are read: https://docs.h5py.org/en/latest/strings.html I am not sure if this has to be fixed in xarray or in h5necdf. I'd be happy if someone else took this one.

Failed tests:

Example failure:

```python traceback

assert_allclose(original, computed) E AssertionError: Left and right Dataset objects are not close E
E Differing coordinates: E L * dim3 (dim3) <U1 'a' 'b' 'c' 'd' 'e' 'f' 'g' 'h' 'i' 'j' E R * dim3 (dim3) object b'a' b'b' b'c' b'd' b'e' b'f' b'g' b'h' b'i' b'j'< ```