When dask is missing we get the following error message:

python-traceback ValueError: unrecognized chunk manager dask - must be one of: []

this could be confusing - the error message seems geared towards a typo in the requested manager. However, I think it's much more likely that a chunk manager is just not installed. I tried to update the error message - happy to get feedback.

Supress some more internal warnings in the test suite.

• [ ] Adresses points 2 and 3 and closes #8612
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Probably not ready for review yet.

What is your issue?

I looked a bit into the frequency string update & found 3 issues we could improve upon.

1. Apart from "M", pandas also deprecated "Y", and "Q", in favor of "YE" and "QE". (And they are discussing renaming "MS" to "MB"). Should we do the same?

2. Should we translate the new freq strings to the old ones if pandas < 2.2 is installed? Otherwise we get the following situation: python import xarray as xr xr.date_range("1600-02-01", periods=3, freq="M") # deprecation warning xr.date_range("1600-02-01", periods=3, freq="ME") # ValueError: Invalid frequency: ME

3. date_range_like can emit deprecation warnings without a way to mitigate them if pandas < 2.2 is installed. (When a DatetimeIndex) is passed. Could be nice to translate the old freq string to the new one without a warning.

I have played around with 2. and 3. and can open a PR if you are on board.

@spencerkclark @aulemahal

• pandas-dev/pandas#55792
• pandas-dev/pandas#55553
• pandas-dev/pandas#56840

• [x] Closes #7104
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Unstacking non-unique MultiIndex can lead to silent data loss, so we raise an error.

What happened?

As per: https://github.com/pydata/xarray/pull/8246#discussion_r1340357405

Passing a negative step in a slice to select a non-chunked zarr-backed datasets raises an error.

What did you expect to happen?

zarr should allow negative step (probably?)

Minimal Complete Verifiable Example

```Python import xarray as xr

create a zarr dataset

air = xr.tutorial.open_dataset("air_temperature") air.to_zarr("test.zarr")

ds = xr.open_dataset("test.zarr", engine="zarr") ds.air[::-1, ].load()

note that this works if the dataset is backed by dask

ds_dask = xr.open_dataset("test.zarr", engine="zarr", chunks="auto") ds_dask.air[::-1, ].load() ```

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python File ~/code/xarray/xarray/core/parallelcompat.py:93, in guess_chunkmanager(manager) 91 if isinstance(manager, str): 92 if manager not in chunkmanagers: ---> 93 raise ValueError( 94 f"unrecognized chunk manager {manager} - must be one of: {list(chunkmanagers)}" 95 ) 97 return chunkmanagers[manager] 98 elif isinstance(manager, ChunkManagerEntrypoint): 99 # already a valid ChunkManager so just pass through

ValueError: unrecognized chunk manager dask - must be one of: [] ```

Anything else we need to know?

The error comes from https://github.com/zarr-developers/zarr-python/blob/6ec746ef1242dd9fec26b128cc0b3455d28ad6f0/zarr/indexing.py#L174 so it would need an upstream fix first.

cc @dcherian is this what you had in mind?

Environment

Fix a warning from ruff concerning the config:

warning: The top-level linter settings are deprecated in favour of their counterparts in the lint section. Please update the following options in pyproject.toml: - 'extend-safe-fixes' -> 'lint.extend-safe-fixes' - 'per-file-ignores' -> 'lint.per-file-ignores'

What happened?

ds.weighted(weights).mean(dims) errors when reducing over a dimension that is neither on the weights nor on the variable.

What did you expect to happen?

This used to work and was "broken" by #8606. However, we may want to fix this by ignoring (?) those data vars instead (#7027).

Minimal Complete Verifiable Example

```Python import xarray as xr

ds = xr.Dataset({"a": (("y", "x"), [[1, 2]]), "scalar": 1}) weights = xr.DataArray([1, 2], dims="x")

ds.weighted(weights).mean("y") ```

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [ ] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

```Python ValueError Traceback (most recent call last) Cell In[1], line 6 3 ds = xr.Dataset({"a": (("y", "x"), [[1, 2]]), "scalar": 1}) 4 weights = xr.DataArray([1, 2], dims="x") ----> 6 ds.weighted(weights).mean("y")

File ~/code/xarray/xarray/util/deprecation_helpers.py:115, in _deprecate_positional_args.<locals>._decorator.<locals>.inner(args, kwargs) 111 kwargs.update({name: arg for name, arg in zip_args}) 113 return func(args[:-n_extra_args], kwargs) --> 115 return func(*args, kwargs)

File ~/code/xarray/xarray/core/weighted.py:497, in Weighted.mean(self, dim, skipna, keep_attrs) 489 @_deprecate_positional_args("v2023.10.0") 490 def mean( 491 self, (...) 495 keep_attrs: bool | None = None, 496 ) -> T_Xarray: --> 497 return self._implementation( 498 self._weighted_mean, dim=dim, skipna=skipna, keep_attrs=keep_attrs 499 )

File ~/code/xarray/xarray/core/weighted.py:558, in DatasetWeighted._implementation(self, func, dim, kwargs) 555 def _implementation(self, func, dim, kwargs) -> Dataset: 556 self._check_dim(dim) --> 558 return self.obj.map(func, dim=dim, **kwargs)

File ~/code/xarray/xarray/core/dataset.py:6924, in Dataset.map(self, func, keep_attrs, args, kwargs) 6922 if keep_attrs is None: 6923 keep_attrs = _get_keep_attrs(default=False) -> 6924 variables = { 6925 k: maybe_wrap_array(v, func(v, *args, kwargs)) 6926 for k, v in self.data_vars.items() 6927 } 6928 if keep_attrs: 6929 for k, v in variables.items():

File ~/code/xarray/xarray/core/dataset.py:6925, in <dictcomp>(.0) 6922 if keep_attrs is None: 6923 keep_attrs = _get_keep_attrs(default=False) 6924 variables = { -> 6925 k: maybe_wrap_array(v, func(v, args, *kwargs)) 6926 for k, v in self.data_vars.items() 6927 } 6928 if keep_attrs: 6929 for k, v in variables.items():

File ~/code/xarray/xarray/core/weighted.py:286, in Weighted._weighted_mean(self, da, dim, skipna) 278 def _weighted_mean( 279 self, 280 da: T_DataArray, 281 dim: Dims = None, 282 skipna: bool | None = None, 283 ) -> T_DataArray: 284 """Reduce a DataArray by a weighted mean along some dimension(s).""" --> 286 weighted_sum = self._weighted_sum(da, dim=dim, skipna=skipna) 288 sum_of_weights = self._sum_of_weights(da, dim=dim) 290 return weighted_sum / sum_of_weights

File ~/code/xarray/xarray/core/weighted.py:276, in Weighted._weighted_sum(self, da, dim, skipna) 268 def _weighted_sum( 269 self, 270 da: T_DataArray, 271 dim: Dims = None, 272 skipna: bool | None = None, 273 ) -> T_DataArray: 274 """Reduce a DataArray by a weighted sum along some dimension(s).""" --> 276 return self._reduce(da, self.weights, dim=dim, skipna=skipna)

File ~/code/xarray/xarray/core/weighted.py:231, in Weighted._reduce(da, weights, dim, skipna) 227 da = da.fillna(0.0) 229 # dot does not broadcast arrays, so this avoids creating a large 230 # DataArray (if weights has additional dimensions) --> 231 return dot(da, weights, dim=dim)

File ~/code/xarray/xarray/util/deprecation_helpers.py:140, in deprecate_dims.<locals>.wrapper(args, kwargs) 132 emit_user_level_warning( 133 "The dims argument has been renamed to dim, and will be removed " 134 "in the future. This renaming is taking place throughout xarray over the " (...) 137 PendingDeprecationWarning, 138 ) 139 kwargs["dim"] = kwargs.pop("dims") --> 140 return func(args, **kwargs)

File ~/code/xarray/xarray/core/computation.py:1885, in dot(dim, arrays, *kwargs) 1883 dim = tuple(d for d, c in dim_counts.items() if c > 1) 1884 else: -> 1885 dim = parse_dims(dim, all_dims=tuple(all_dims)) 1887 dot_dims: set[Hashable] = set(dim) 1889 # dimensions to be parallelized

File ~/code/xarray/xarray/core/utils.py:1046, in parse_dims(dim, all_dims, check_exists, replace_none) 1044 dim = (dim,) 1045 if check_exists: -> 1046 _check_dims(set(dim), set(all_dims)) 1047 return tuple(dim)

File ~/code/xarray/xarray/core/utils.py:1131, in _check_dims(dim, all_dims) 1129 if wrong_dims: 1130 wrong_dims_str = ", ".join(f"'{d!s}'" for d in wrong_dims) -> 1131 raise ValueError( 1132 f"Dimension(s) {wrong_dims_str} do not exist. Expected one or more of {all_dims}" 1133 )

ValueError: Dimension(s) 'y' do not exist. Expected one or more of {'x'} ```

Anything else we need to know?

No response

Environment

Newest main (i.e. 2024.01)

• [ ] Closes #xxxx
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

This allows negative freq strings as discussed in https://github.com/pydata/xarray/pull/8627#issuecomment-1905981660 Deciding which tests to update was not easy.

The pandas _generate_range function was moved to https://github.com/pandas-dev/pandas/blob/3c96b8ff6d399fbec8d4d533e8e8618c592bb64b/pandas/core/arrays/datetimes.py#L2725 They no longer rollback the end. I had to remove this as well such that the following are eqivalent:

python xr.date_range("2001", "2000", freq="-1YE", calendar="noleap") pd.date_range("2001", "2000", freq="-1YE")

I am slightly nervous about this but all the tests still pass...

Once again cc @spencerkclark

use ruff.flake8-tidy-imports to enforce absolute imports

• https://github.com/MarcoGorelli/absolufy-imports has been archived (no reason given)
• removes a pre-commit hook which should make it faster locally

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

I think we should use extend-exclude instead of exclude for ruff. We can then also remove ".eggs" as this is in the default.

From https://docs.astral.sh/ruff/settings/#exclude:

Note that you'll typically want to use extend-exclude to modify the excluded paths.

Default value: [".bzr", ".direnv", ".eggs", ".git", ".git-rewrite", ".hg", ".mypy_cache", ".nox", ".pants.d", ".pytype", ".ruff_cache", ".svn", ".tox", ".venv", "__pypackages__", "_build", "buck-out", "build", "dist", "node_modules", "venv"]

(I really dislike how github formats toml files... What would be the correct syntax, then?)

I realized that the return value of infer_freq was not updated. #8627 will try to suppress all warnings in the test suite, so this is just the minimal PR.

Sorry for all the spam @spencerkclark

• [x] Adresses point 1 of #8612
• [x] Fixes one of the failures in #8623
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

This renames the frequency string "Y" (formerly "A") to "YE" to achieve pandas parity. It could be better to wait for the conclusion of pandas-dev/pandas#56840 before doing this (but fixing the related failure in #8623 seemed a good reason as any to do it know).

Let me know what you think @spencerkclark @aulemahal

• [x] Closes #7298
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Otherwise da.indexes and the html repr raise a ValueError. I first had "<undefined>" but I think None is better. cc @spencerkclark @keewis

What happened?

The html repr of a cftime array wants to display the "calendar", which it cannot if it is empty.

What did you expect to happen?

No error.

Minimal Complete Verifiable Example

```Python import numpy as np import xarray as xr

data_obs = np.random.randn(3) time_obs = xr.date_range("2000-01-01", periods=3, freq="YS", calendar="noleap")

obs = xr.DataArray(data_obs, coords={"time": time_obs})

o = obs[:0]

xr.core.formatting_html.array_repr(o) ```

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python ValueError Traceback (most recent call last) Input In [1], in <cell line: 12>() 8 obs = xr.DataArray(data_obs, coords={"time": time_obs}) 10 o = obs[:0] ---> 12 xr.core.formatting_html.array_repr(o)

File ~/code/xarray/xarray/core/formatting_html.py:318, in array_repr(arr) 316 if hasattr(arr, "xindexes"): 317 indexes = _get_indexes_dict(arr.xindexes) --> 318 sections.append(index_section(indexes)) 320 sections.append(attr_section(arr.attrs)) 322 return _obj_repr(arr, header_components, sections)

File ~/code/xarray/xarray/core/formatting_html.py:195, in _mapping_section(mapping, name, details_func, max_items_collapse, expand_option_name, enabled) 188 expanded = _get_boolean_with_default( 189 expand_option_name, n_items < max_items_collapse 190 ) 191 collapsed = not expanded 193 return collapsible_section( 194 name, --> 195 details=details_func(mapping), 196 n_items=n_items, 197 enabled=enabled, 198 collapsed=collapsed, 199 )

File ~/code/xarray/xarray/core/formatting_html.py:155, in summarize_indexes(indexes) 154 def summarize_indexes(indexes): --> 155 indexes_li = "".join( 156 f"

{indexes_li}

File ~/code/xarray/xarray/core/formatting_html.py:156, in <genexpr>(.0) 154 def summarize_indexes(indexes): 155 indexes_li = "".join( --> 156 f"

{indexes_li}

File ~/code/xarray/xarray/core/formatting_html.py:140, in summarize_index(coord_names, index) 138 index_id = f"index-{uuid.uuid4()}" 139 preview = escape(inline_index_repr(index)) --> 140 details = short_index_repr_html(index) 142 data_icon = _icon("icon-database") 144 return ( 145 f"

File ~/code/xarray/xarray/core/formatting_html.py:132, in short_index_repr_html(index) 129 if hasattr(index, "repr_html"): 130 return index.repr_html() --> 132 return f"

{escape(repr(index))}

File ~/code/xarray/xarray/core/indexes.py:547, in PandasIndex.repr(self) 546 def repr(self): --> 547 return f"PandasIndex({repr(self.index)})"

File ~/code/xarray/xarray/coding/cftimeindex.py:353, in CFTimeIndex.repr(self) 345 end_str = format_times( 346 self.values[-REPR_ELLIPSIS_SHOW_ITEMS_FRONT_END:], 347 display_width, 348 offset=offset, 349 first_row_offset=offset, 350 ) 351 datastr = "\n".join([front_str, f"{' '*offset}...", end_str]) --> 353 attrs_str = format_attrs(self) 354 # oneliner only if smaller than display_width 355 full_repr_str = f"{klass_name}([{datastr}], {attrs_str})"

File ~/code/xarray/xarray/coding/cftimeindex.py:272, in format_attrs(index, separator) 267 def format_attrs(index, separator=", "): 268 """Format attributes of CFTimeIndex for repr.""" 269 attrs = { 270 "dtype": f"'{index.dtype}'", 271 "length": f"{len(index)}", --> 272 "calendar": f"'{index.calendar}'", 273 "freq": f"'{index.freq}'" if len(index) >= 3 else None, 274 } 276 attrs_str = [f"{k}={v}" for k, v in attrs.items()] 277 attrs_str = f"{separator}".join(attrs_str)

File ~/code/xarray/xarray/coding/cftimeindex.py:698, in CFTimeIndex.calendar(self) 695 """The calendar used by the datetimes in the index.""" 696 from .times import infer_calendar_name --> 698 return infer_calendar_name(self)

File ~/code/xarray/xarray/coding/times.py:374, in infer_calendar_name(dates) 371 return sample.calendar 373 # Error raise if dtype is neither datetime or "O", if cftime is not importable, and if element of 'O' dtype is not cftime. --> 374 raise ValueError("Array does not contain datetime objects.")

ValueError: Array does not contain datetime objects. ```

Anything else we need to know?

Bisected to 7379923de756a2bcc59044d548f8ab7a68b91d4e use _repr_inline_ for indexes that define it.

Environment

By chance I saw that in #4700 the same code block was added twice. I think this can be removed.

cc @andersy005

• [x] Closes #8367
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Uses max with a default, which work with empty iterators, in contrast to if col_items else 0.

What happened?

da.xindexes or da.indexes raises an error when trying to generate the repr if there are no coords (indexes)

What did you expect to happen?

Displaying an empty Mappable?

Minimal Complete Verifiable Example

Python xr.DataArray([3, 5]).indexes xr.DataArray([3, 5]).xindexes

MVCE confirmation

• [x] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [x] Complete example — the example is self-contained, including all data and the text of any traceback.
• [x] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [x] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [x] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

```Python Out[9]: --------------------------------------------------------------------------- ValueError Traceback (most recent call last) File ~/.conda/envs/xarray_dev/lib/python3.10/site-packages/IPython/core/formatters.py:708, in PlainTextFormatter.call(self, obj) 701 stream = StringIO() 702 printer = pretty.RepresentationPrinter(stream, self.verbose, 703 self.max_width, self.newline, 704 max_seq_length=self.max_seq_length, 705 singleton_pprinters=self.singleton_printers, 706 type_pprinters=self.type_printers, 707 deferred_pprinters=self.deferred_printers) --> 708 printer.pretty(obj) 709 printer.flush() 710 return stream.getvalue()

File ~/.conda/envs/xarray_dev/lib/python3.10/site-packages/IPython/lib/pretty.py:410, in RepresentationPrinter.pretty(self, obj) 407 return meth(obj, self, cycle) 408 if cls is not object \ 409 and callable(cls.dict.get('repr')): --> 410 return _repr_pprint(obj, self, cycle) 412 return _default_pprint(obj, self, cycle) 413 finally:

File ~/.conda/envs/xarray_dev/lib/python3.10/site-packages/IPython/lib/pretty.py:778, in repr_pprint(obj, p, cycle) 776 """A pprint that just redirects to the normal repr function.""" 777 # Find newlines and replace them with p.break() --> 778 output = repr(obj) 779 lines = output.splitlines() 780 with p.group():

File ~/code/xarray/xarray/core/indexes.py:1659, in Indexes.repr(self) 1657 def repr(self): 1658 indexes = formatting._get_indexes_dict(self) -> 1659 return formatting.indexes_repr(indexes)

File ~/code/xarray/xarray/core/formatting.py:474, in indexes_repr(indexes, max_rows) 473 def indexes_repr(indexes, max_rows: int | None = None) -> str: --> 474 col_width = _calculate_col_width(chain.from_iterable(indexes)) 476 return _mapping_repr( 477 indexes, 478 "Indexes", (...) 482 max_rows=max_rows, 483 )

File ~/code/xarray/xarray/core/formatting.py:341, in _calculate_col_width(col_items) 340 def _calculate_col_width(col_items): --> 341 max_name_length = max(len(str(s)) for s in col_items) if col_items else 0 342 col_width = max(max_name_length, 7) + 6 343 return col_width

ValueError: max() arg is an empty sequence ```

Anything else we need to know?

No response

Environment

I really should not open this can of worms but per https://github.com/pydata/xarray/issues/4450#issuecomment-697507489:

I'm always confused about whether ds.groupby(..., keep_attrs=True).mean() or ds.groupby(...).mean(keep_attrs=True) is correct. (similarly for rolling, coarsen etc.)

Also as I try to fix the keep_attr behavior in #4510 it would be good to know where they should go. So I tried to figure out how this is currently handled and found the following:

ds.xxx(keep_attrs=True).yyy() - all fixed

ds.xxx().yyy(keep_attrs=True) - coarsen (fixed in #5227) - groupby - groupby_bin - resample - rolling (adjusted in #4510) - rolling_exp (fixed in #4592) - weighted

So the working consensus seems to be to to ds.xxx().yyy(keep_attrs=True) - any comments on that?

(Edit: looking at this it is only half as bad, "only" coarsen, rolling (#4510), and rolling_exp would need to be fixed.)

Detailed analysis

edit: moved rolling after #4510, moved rolling_exp after #4592 and coarsen after #5227

What happened?

I tried a DataArray with more than one dimension coordinate. Unfortunately xr.align fails, which disallows any arithmetic operation - even when the coords are exactly the same.

What did you expect to happen?

No response

Minimal Complete Verifiable Example

```Python import numpy as np import xarray as xr

data = np.arange(12).reshape(3, 4)

y = [10, 20, 30] s = ["a", "b", "c"]

x = [1, 2, 3, 4]

da = xr.DataArray(data, dims=("y", "x"), coords={"x": x, "y": y, "s": ("y", s)}) da = da.set_xindex("s")

xr.align(da, da.y) # errors da + da # errors da + da.x # errors ```

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [ ] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

```Python

ValueError Traceback (most recent call last) /home/mathause/code/mesmer/devel/prepare_for_surfer.ipynb Cell 28 line 1 12 da = xr.DataArray(data, dims=("y", "x"), coords={"x": x, "y": y, "s": ("y", s)}) 13 da = da.set_xindex("s") ---> 15 xr.align(da, da.y) # errors 17 da + da.x # errors

File ~/.conda/envs/mesmer_dev/lib/python3.9/site-packages/xarray/core/alignment.py:888, in align(join, copy, indexes, exclude, fill_value, *objects) 692 """ 693 Given any number of Dataset and/or DataArray objects, returns new 694 objects with aligned indexes and dimension sizes. ref='~/.conda/envs/mesmer_dev/lib/python3.9/site-packages/xarray/core/alignment.py:0'>0;32m (...) 878 879 """ 880 aligner = Aligner( 881 objects, 882 join=join, ref='~/.conda/envs/mesmer_dev/lib/python3.9/site-packages/xarray/core/alignment.py:0'>0;32m (...) 886 fill_value=fill_value, 887 ) --> 888 aligner.align() 889 return aligner.results

File ~/.conda/envs/mesmer_dev/lib/python3.9/site-packages/xarray/core/alignment.py:573, in Aligner.align(self) 571 self.find_matching_indexes() 572 self.find_matching_unindexed_dims() --> 573 self.assert_no_index_conflict() 574 self.align_indexes() 575 self.assert_unindexed_dim_sizes_equal()

File ~/.conda/envs/mesmer_dev/lib/python3.9/site-packages/xarray/core/alignment.py:318, in Aligner.assert_no_index_conflict(self) 314 if dup: 315 items_msg = ", ".join( 316 f"{k!r} ({v} conflicting indexes)" for k, v in dup.items() 317 ) --> 318 raise ValueError( 319 "cannot re-index or align objects with conflicting indexes found for " 320 f"the following {msg}: {items_msg}\n" 321 "Conflicting indexes may occur when\n" 322 "- they relate to different sets of coordinate and/or dimension names\n" 323 "- they don't have the same type\n" 324 "- they may be used to reindex data along common dimensions" 325 )

ValueError: cannot re-index or align objects with conflicting indexes found for the following dimensions: 'y' (2 conflicting indexes) Conflicting indexes may occur when - they relate to different sets of coordinate and/or dimension names - they don't have the same type - they may be used to reindex data along common dimensions ```

Anything else we need to know?

No response

Environment

What happened?

Running an in-memory groupby operation took much longer than expected. Turning off flox fixed this - but I don't think that's the idea ;-)

What did you expect to happen?

flox to be at least on par with our naive implementation

Minimal Complete Verifiable Example

```Python import numpy as np import xarray as xr

arr = np.random.randn(10, 10, 36530) time = xr.date_range("2000", periods=30365, calendar="noleap") da = xr.DataArray(arr, dims=("y", "x", "time"), coords={"time": time})

using max

print("max:") xr.set_options(use_flox=True) %timeit da.groupby("time.year").max("time") %timeit da.groupby("time.year").max("time", engine="flox")

xr.set_options(use_flox=False) %timeit da.groupby("time.year").max("time")

as reference

%timeit [da.sel(time=str(year)).max("time") for year in range(2000, 2030)]

using mean

print("mean:") xr.set_options(use_flox=True) %timeit da.groupby("time.year").mean("time") %timeit da.groupby("time.year").mean("time", engine="flox")

xr.set_options(use_flox=False) %timeit da.groupby("time.year").mean("time")

as reference

%timeit [da.sel(time=str(year)).mean("time") for year in range(2000, 2030)] ```

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python max: 158 ms ± 4.41 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) 28.1 ms ± 318 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) 11.5 ms ± 52.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

mean: 95.6 ms ± 10.8 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) 34.8 ms ± 2.88 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) 15.2 ms ± 232 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) ```

Anything else we need to know?

No response

Environment

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

This makes many arguments keyword-only, except for dim to avoid da.weighted(...).mean("lat", "lon") (i.e. da.weighted(...).mean(dim="lat", skipna="lon")) which silently does the wrong thing. I am sure I forgot some and for some I was unsure so I left them as is.

Question: do we want an deprecation cycle? Currently it just errors for da.weighted(...).mean("dim", True). Might be nice to do it, however, @dcherian if I am not mistaken you did this without a deprecation in #5950, e.g. for da.mean etc.?

python import numpy as np import xarray as xr air = xr.tutorial.open_dataset("air_temperature") wgt = np.cos(np.deg2rad(air.lat)) air.weighted(wgt).mean("lat", "lon")

• [ ] Towards #8239
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

This partly updates the pytest config as suggested in #8239 and un-xfails some tests (or xfails the tests more precisely).

See https://github.com/pydata/xarray/issues/8239#issuecomment-1739363809 for why we cannot exactly follow the suggestions given in #8239

In #4592 I had the issue that mypy did not raise an error on a missing method:

```python from xarray.core.common import DataWithCoords

hasattr(xr.core.common.DataWithCoords, "reduce") # -> False

def test(x: "DataWithCoords"): x.reduce() # mypy does not error ```

This is because DataWithCoords implements __getattr__:

```python

class A: pass

class B: def getattr(self, name): ...

def testA(x: "A"): x.reduce() # mypy errors

def testB(x: "B"): x.reduce() # mypy does not error ```

The solution seems to be to not typecheck __getattr__ (see https://github.com/python/mypy/issues/6251#issuecomment-457287161):

```python from typing import no_type_check

class C: @no_type_check def getattr(self, name): ...

def testC(x: "C"): x.reduce() # mypy errors ```

The only __getattr__ within xarray is here:

https://github.com/pydata/xarray/blob/17358922d480c038e66430735bf4c365a7677df8/xarray/core/common.py#L221

Using @no_type_check leads to 24 errors and not all of them can be trivially solved. E.g. DataWithCoords wants of use self.isel but does not implement the method. The solution is probably to add isel to DataWithCoords as an ABC or using NotImplemented.

Thoughts?

All errors

Edit: one problem is certainly the method injection, as mypy cannot detect those types.

When I manually specify the units attribute for time, and then save the Dataset to_netcdf the string is truncated. See exaple

import pandas as pd
import xarray as xr

time = pd.date_range('2000-01-01', '2000-01-31', freq='6h')
ds = xr.Dataset(coords=dict(time=time))

units = 'days since 1975-01-01 00:00:00'
calendar = 'gregorian'
encoding=dict(time=dict(units=units, calendar=calendar))

ds.to_netcdf('test.nc', format='NETCDF4_CLASSIC', encoding=encoding)

! ncdump -h test.nc
# time:units = "days since 1975-01-01" ;

Some programs seem to require the hours to be present to interpret the time properly (e.g. panoply). When specifying the hour, a 'T' is added.

units = 'days since 1975-01-01 01:00:00'

! ncdump -h test.nc
# time:units = "days since 1975-01-01T01:00:00" ;

When xarray defines the time.units it works fine.

ds = xr.Dataset(coords=dict(time=time))
ds.to_netcdf('test.nc', format='NETCDF4_CLASSIC',)

! ncdump -h test.nc
# time:units = "hours since 2000-01-01 00:00:00" ;

xarray version 0.9.6

• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

I found the error message in ds.to_stacked_array confusing, so I tried to make it clearer. Also renames some if the internal symbols (so should have no user facing change).

What is your issue?

Applying e.g. ds.weighted(weights).mean() applies the operation over all DataArray objects - even if they don't have the dimensions over which it is applied (or is a scalar variable). I don't think this is wanted.

```python import xarray as xr

air = xr.tutorial.open_dataset("air_temperature") air.attrs = {}

add variable without dims

air["foo"] = 5

print("resample") print(air.resample(time="MS").mean(dim="time").foo.dims)

print("groupby") print(air.groupby("time.year").mean(dim="time").foo.dims)

print("weighted") print(air.weighted(weights=air.time.dt.year).mean("lat").foo.dims)

print("where") print(air.where(air.air > 5).foo.dims) ```

Results resample ('time',) groupby ('year',) weighted ('time',)

Related #6952 - I am sure there are other issues, but couldn't find them quickly...

rolling and coarsen don't seem to do this.

What happened?

A cftime variable returns a timedelta64[ns] when calling diff / + / - and it can then not be added/ subtracted from the original data.

What did you expect to happen?

We can add cftime timedeltas.

Minimal Complete Verifiable Example

```Python import xarray as xr

air = xr.tutorial.open_dataset("air_temperature", use_cftime=True)

air.time + air.time.diff("time") / 2 ```

MVCE confirmation

• [x] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [x] Complete example — the example is self-contained, including all data and the text of any traceback.
• [x] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [x] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

Python air.time.variable.values[1:] - air.time.variable.values[:-1] returns array([datetime.timedelta(seconds=21600), ...]) but then Python xr.Variable(("time",), np.array([datetime.timedelta(0)])) returns a dtype='timedelta64[ns]' array.

Anything else we need to know?

• See upstream PR: xarray-contrib/cf-xarray#441
• Similar to #7381 (but I don't think it's the same issue, feel free to close if you disagree)
• That might need a special data type for timedeltas of cftime.Datetime objects, or allowing to add 'timedelta64[ns]' to cftime.Datetime objects
• The casting comes from

https://github.com/pydata/xarray/blob/d8ec3a3f6b02a8b941b484b3d254537af84b5fde/xarray/core/variable.py#L366

https://github.com/pydata/xarray/blob/d8ec3a3f6b02a8b941b484b3d254537af84b5fde/xarray/core/variable.py#L272

Environment

It would be nice to make weighted work with groupby - e.g. #3935 (comment)

However, it is not entirely clear to me how that should be done. One way would be to do: python da.groupby(...).weighted(weights).mean() this would require that the groupby operation is applied over the weights (how would this be done?) Or should it be

python da.weighted(weights).groupby(...).mean() but this seems less intuitive to me.

Or python da.groupby(..., weights=weights).mean()

Suppress the pkg_resources.namespace_package DeprecationError to make the doctest pass again (similar to #7322). This is reported upstream: pydap/pydap#277 and matplotlib/matplotlib#25244

• [x] Closes #6092
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

One change is that it is no longer possible to compare to a string, i.e. version.parse(xr.__version__) < "0.20.0" errors.

As mentioned in #6092 there are 3 options - if there is a preference I am happy to update this PR.

```python from distutils.version import LooseVersion from packaging import version

LooseVersion(xr.version) version.parse(xr.version) version.Version(xr.version)

currently:

if LooseVersion(mod.version) < LooseVersion(minversion): pass

options:

if version.parse(mod.version) < version.parse(minversion): pass

if version.Version(mod.version) < version.Version(minversion): pass

if Version(mod.version) < Version(minversion): pass ```

• [x] Closes #7322

This PR renames the da and ds fixtures (to da_fixture and ds_fixture) and moves them to conftest.py. This allows to remove the flake8 error suppression for the tests and seems more how the fixtures are intended to be used (from the pytest side). I think the name changes makes it a bit more obvious what happens but moving them to may make it a bit less obvious (if you don't know where to look).

Removing the flake8 error ignores also unearthed some unused imports:

https://github.com/pydata/xarray/blob/787a96c15161c9025182291b672b3d3c5548a6c7/setup.cfg#L155-L156

(What I actually wanted to do is move the tests for rolling to it's own file - but I think it makes sense to do this first.)

I forgot to remove some debug print statements in #6910 - thanks for noting @shoyer & @dcherian

flake8 v6 now errors on inline comments in the config file. I don't like it but oh well...

We are currently not testing with the newest version of netCDF4 because it is incompatible with pynio (the newest version is 1.5.4, we are at 1.5.3). This is unlikely to be fixed, see conda-forge/pynio-feedstock#90.

Therefore we need to think how to setup the tests so we use the newest version of netCDF4. Maybe just remove it from py38.yml?

And long term what to do with the pynio backend? Deprecate? Move to an external repo?

What happened?

xr.core.dataset.DataVariables assumes that everything that is in ds._dataset._variables and not in self._dataset._coord_names is a "data variable". However, since the index refactor we can end up with more _coord_names than _variables which breaks a number of stuff (e.g. the repr).

What did you expect to happen?

Well it seems this assumption is now wrong.

Minimal Complete Verifiable Example

Python ds = xr.Dataset(coords={"a": ("x", [1, 2, 3]), "b": ("x", ['a', 'b', 'c'])}) ds.set_index(z=['a', 'b']).reset_index("z", drop=True)

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

Python ValueError: __len__() should return >= 0

Anything else we need to know?

The error comes from here

https://github.com/pydata/xarray/blob/63ba862d03c8d0cd8b44d2071bc360e9fed4519d/xarray/core/dataset.py#L368

Bisected to #5692 - which probably does not help too much.

Environment

In #4864 I added a a decorator for the xarray.testing.assert_* functions to ensure warnings that were to errors (pytest.mark.filterwarnings("error")) do not error in assert_* (see https://github.com/pydata/xarray/pull/4760#issuecomment-774101639). As a solution I added

https://github.com/pydata/xarray/blob/5470d933452d88deb17cc9294a164c4a03f55dec/xarray/testing.py#L32

However, this is sub-optimal because this now removes all ignore filters! As dask stuff only gets evaluated in assert_* filters like warnings.filterwarnings("ignore", "Mean of empty slice") don't work for dask arrays!

I thought of setting

python warnings.simplefilter("ignore")

but this could suppress warnings we want to keep.

So now I remove all "error" warning filters and keep the rest. Note that the original filters get restored after with warnings.catch_warnings():. ().

I am not sure I expressed myself very clearly... let me know and I can try again. @keewis you had a look at #4864 maybe you can review this PR as well?

I tried to use the _deprecate_positional_args decorator from #6934 & it turns out that it still had some errors - passing on the arguments did not work properly in certain cases... I now added tests for this as well (which I should have done in the first place...).

5844 excluded pydap from our tests - but the new version has been released in the meantime (on conda not on pypi, though, pydap/pydap#268) - so let's see if this still works.

• [x] Closes #6960
• [ ] Tests added - tested manually

Get rid of some warnings in the tests.

• [x] Closes #6932

@zmoon - obviously it would be nice if we had a linter for this but this is for another time.

• [x] Supersedes #6403, see also #5531
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Adds a helper function to deprecate positional arguments. IMHO this offers a good trade-off between magic and complexity. (As mentioned this was adapted from scikit-learn).

edit: I suggest to actually deprecate positional arguments in another PR.

fix some typos (using codespell). Called using:

bash codespell --skip=".git,.mypy_cache,*.tex" --ignore-words-list coo,nd,inferrable,hist,ND,splitted,soler,slowy,ba,ser,nin,te,fo -w -i 3

What happened?

We generally type dimensions as:

python dims: Hashable | Iterable[Hashable]

However, this is in conflict with passing a tuple of independent dimensions to a method - e.g. da.mean(("x", "y")) because a tuple is also hashable.

Also mypy requires an isinstance(dims, Hashable) check when typing a function. We use an isinstance(dims, str) check in many places to wrap a single dimension in a list. Changing this to isinstance(dims, Hashable) will change the behavior for tuples.

What did you expect to happen?

In the community call today we discussed to change this to

python dims: str | Iterable[Hashable]

i.e. if a single dim is passed it has to be a string and wrapping it in a list is a convenience function. Special use cases with Hashable types should be wrapped in a Iterable by the user. This probably best reflects the current state of the repo (dims = [dims] if isinstance(dims, str) else dims).

The disadvantage could be that it is a bit more difficult to explain in the docstrings?

@shoyer - did I get this right from the discussion?

Other options

1. Require str as dimension names.

This could be too restrictive. @keewis mentioned that tuple dimension names are already used somwehere in the xarray repo. Also we discussed in another issue or PR (which I cannot find right know) that we want to keep allowing Hashable.

1. Disallow passing tuples (only allow tuples if a dimension is a tuple), require lists to pass several dimensions.

This is too restrictive in the other direction and will probably lead to a lot of downstream troubles. Naming a single dimension with a tuple will be a very rare case, in contrast to passing several dimension names as a tuple.

1. Special case tuples. We could potentially check if dims is a tuple and if there are any dimension names consisting of a tuple. Seems more complicated and potentially brittle for probably small gains (IMO).

Minimal Complete Verifiable Example

No response

Relevant log output

No response

Anything else we need to know?

• We need to check carefully where general Hashable are really allowed. E.g. dims of a DataArray are typed as

https://github.com/pydata/xarray/blob/e056cacdca55cc9d9118c830ca622ea965ebcdef/xarray/core/dataarray.py#L380

but tuples are not actually allowed:

```python import xarray as xr xr.DataArray([1], dims=("x", "y"))

ValueError: different number of dimensions on data and dims: 1 vs 2

xr.DataArray([1], dims=[("x", "y")])

TypeError: dimension ('x', 'y') is not a string

```

• We need to be careful typing functions where only one dim is allowed, e.g. xr.concat, which should probably set dim: Hashable (and make sure it works).
• Do you have examples for other real-world hashable types except for str and tuple? (Would be good for testing purposes).

Environment

N/A

polyfit currently only allows to fit along a dimension and not along a non-dimension coordinate (or a virtual coordinate)

Example: ```python da = xr.DataArray( [1, 3, 2], dims=["x"], coords=dict(x=["a", "b", "c"], y=("x", [0, 1, 2])) )

print(da)

da.polyfit("y", 1) Output:python <xarray.DataArray (x: 3)> array([1, 3, 2]) Coordinates: * x (x) <U1 'a' 'b' 'c' y (x) int64 0 1 2

KeyError Traceback (most recent call last) <ipython-input-80-9bb2dacf50f7> in <module> 5 print(da) 6 ----> 7 da.polyfit("y", 1)

~/.conda/envs/ipcc_ar6/lib/python3.7/site-packages/xarray/core/dataarray.py in polyfit(self, dim, deg, skipna, rcond, w, full, cov) 3507 """ 3508 return self._to_temp_dataset().polyfit( -> 3509 dim, deg, skipna=skipna, rcond=rcond, w=w, full=full, cov=cov 3510 ) 3511

~/.conda/envs/ipcc_ar6/lib/python3.7/site-packages/xarray/core/dataset.py in polyfit(self, dim, deg, skipna, rcond, w, full, cov) 6005 skipna_da = skipna 6006 -> 6007 x = get_clean_interp_index(self, dim, strict=False) 6008 xname = "{}_".format(self[dim].name) 6009 order = int(deg) + 1

~/.conda/envs/ipcc_ar6/lib/python3.7/site-packages/xarray/core/missing.py in get_clean_interp_index(arr, dim, use_coordinate, strict) 246 247 if use_coordinate is True: --> 248 index = arr.get_index(dim) 249 250 else: # string

~/.conda/envs/ipcc_ar6/lib/python3.7/site-packages/xarray/core/common.py in get_index(self, key) 378 """ 379 if key not in self.dims: --> 380 raise KeyError(key) 381 382 try:

KeyError: 'y' ```

Describe the solution you'd like

Would be nice if that worked.

Describe alternatives you've considered

One could just set the non-dimension coordinate as index, e.g.: da = da.set_index(x="y")

Additional context

Allowing this may be as easy as replacing

https://github.com/pydata/xarray/blob/9c85dd5f792805bea319f01f08ee51b83bde0f3b/xarray/core/missing.py#L248

by index = arr[dim] but I might be missing something. Or probably a use_coordinate must be threaded through to get_clean_interp_index (although I am a bit confused by this argument).

This started as a question and I add it as reference. Maybe you have a comment.

There are several ways to calculate time series of seasonal data (starting from monthly or daily data):

```

load libraries

import pandas as pd import matplotlib.pyplot import numpy as np import xarray as xr

Create Example Dataset

time = pd.date_range('2000.01.01', '2010.12.31', freq='M') data = np.random.rand(*time.shape) ds = xr.DataArray(data, coords=dict(time=time))

(1) using resample

ds_res = ds.resample('Q-FEB', 'time') ds_res = ds_res.sel(time=ds_res['time.month'] == 2) ds_res = ds_res.groupby('time.year').mean('time')

(2) this is wrong

ds_season = ds.where(ds['time.season'] == 'DJF').groupby('time.year').mean('time')

(3) using where and rolling

mask other months with nan

ds_DJF = ds.where(ds['time.season'] == 'DJF')

rolling mean -> only Jan is not nan

however, we loose Jan/ Feb in the first year and Dec in the last

ds_DJF = ds_DJF.rolling(min_periods=3, center=True, time=3).mean()

make annual mean

ds_DJF = ds_DJF.groupby('time.year').mean('time')

ds_res.plot(marker='*') ds_season.plot() ds_DJF.plot()

plt.show() ```

(1) The first is to use resample with 'Q-FEB' as argument. This works fine. It does include Jan/ Feb in the first year, and Dec in the last year + 1. If this makes sense can be debated. One case where this does not work is when you have, say, two regions in your data set, for one you want to calculate DJF and for the other you want NovDecJan.

(2) Using 'time.season' is wrong as it combines Jan, Feb and Dec from the same year.

(3) The third uses where and rolling and you lose 'incomplete' seasons. If you replace ds.where(ds['time.season'] == 'DJF') with ds.groupby('time.month').where(summer_months), where summer_months is a boolean array it works also for non-standard 'summers' (or seasons) across the globe.

Code Sample, a copy-pastable example if possible

```python

Your code here

import xarray as xr import numpy as np x = np.arange(10.) da = xr.Dataset(data_vars=dict(data=('dim1', x)), coords=dict(dim1=('dim1', x))) da.to_netcdf('tst.nc')

```

Problem description

Apologies if this was discussed somwhere and it probably does not matter much, but tst.nc has _FillValue although it is not really necessary.

Output of xr.show_versions()

• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

skipna=None did not skip missing values for quantile, inconsistent with other methods. Discovered while testing #6059.

• [x] Closes #6136

Convert the code block in xr.DataArray.from_dict and xr.Dataset.from_dict to doctest/ examples.

Is your feature request related to a problem?

xr.broadcast always does an outer join:

https://github.com/pydata/xarray/blob/de965f342e1c9c5de92ab135fbc4062e21e72453/xarray/core/alignment.py#L702

https://github.com/pydata/xarray/blob/de965f342e1c9c5de92ab135fbc4062e21e72453/xarray/core/alignment.py#L768

This is not how the (default) broadcasting (arithmetic join) works, e.g. the following first does an inner join and then broadcasts:

```python import xarray as xr

da1 = xr.DataArray([[0, 1, 2]], dims=("y", "x"), coords={"x": [0, 1, 2]}) da2 = xr.DataArray([0, 1, 2, 3, 4], dims="x", coords={"x": [0, 1, 2, 3, 4]}) da1 + da2 ```

<xarray.DataArray (y: 1, x: 3)> array([[0, 2, 4]]) Coordinates: * x (x) int64 0 1 2 Dimensions without coordinates: y

Describe the solution you'd like

Add a join argument to xr.broadcast. I would propose to leave the default as is

python def broadcast(*args, exclude=None, join="outer"): args = align(*args, join=join, copy=False, exclude=exclude)

Describe alternatives you've considered

• We could make broadcast respect options -> arithmetic_join but that would be a breaking change and I am not sure how the deprecation should/ would be handled...
• We could leave it as is.

Additional context

• xr.broadcast should not be used often because this is should happen automatically in most cases
• in #6059 I use broadcast because I couldn't get it to work otherwise (maybe there is a better way?). However, the "outer elements" are immediately discarded again - so it's kind of pointless to do an outer join.

```python import numpy as np import xarray as xr

da = xr.DataArray(np.arange(6).reshape(3, 2), coords={"dim_0": [0, 1, 2]}) w = xr.DataArray([1, 1, 1, 1, 1, 1], coords={"dim_0": [0, 1, 2, 4, 5, 6]}) da.weighted(w).quantile(0.5) ```

pytest v7.0.0 no longer want's us to use pytest.warns(None) to test for no warning, so we can use warnings.catch_warnings(record=True) instead.

numpy/numpy#20327 introduces some changes to np.quantile (and related) for the upcoming numpy release (v1.22.0). It renames the interpolation keyword to method and offers some new interpolation methods. This PR does two things

1. it restores compatibility with numpy 1.22

2. it renames the interpolation keyword to method in xarray - this change is not strictly necessary but I thought better to be consistent with numpy

3. [x] Tests added

4. [x] Passes pre-commit run --all-files
5. [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

(Side note in dask.array.percentile the method keyword is used differently from the interpolation keyword (docs). However, xarray does not use the dask function.)

TODO: need to import ArrayLike from npcompat.

• [x] Closes #6186

pydata/bottleneck#378 was merged - so this should work again.

bottleneck cannot be built with python 3.10. See https://github.com/pydata/xarray/actions/runs/1731371015

Fixes or suppresses a number of warnings that turn up in our upstream CI.

pd.Index.is_monotonic is an alias for pd.Index.is_monotonic_increasing and does not stand for pd.Index.is_monotonic_increasing or pd.Index.is_monotonic_decreasing.

Reverts pydata/xarray#6200

And another tiny one... The somewhat ambiguous date string triggers a warning in pandas which makes our doc build fail.

• [x] Closes #5721
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

Fixed as per @shoyer & @spencerkclark suggestion from https://github.com/pydata/xarray/issues/5721#issuecomment-903095007

Now that pandas 1.4 is out it would be good to get this fixed (there are about 5000 warnings in our tests, mostly because of interp, though). Also leads to a warning in our docs which breaks them (although that can also be fixed with an :okwarning: directive).

pandas deprecates the method keyword in Index.get_loc, see pandas-dev/pandas#42269. Therefore we end up with about 5000 warnings in our upstream tests:

FutureWarning: Passing method to Index.get_loc is deprecated and will raise in a future version. Use index.get_indexer([item], method=...) instead

We should fix this before pandas releases because the warning will not be silent (FutureWarning) or ask pandas to give us more time and use a DeprecationWarning at the moment.

@benbovy @spencerkclark

Fixes the doctests for the newest version of cftime. @spencerkclark

This of course means that the doctests will fail for environments with older versions of cftime present. I don't think there is anything we can do.

Thanks for pytest-accept b.t.w @max-sixty

5740 changed PendingDeprecationWarning to FutureWarning - suppress the warnings again in the test suite.

https://github.com/pydata/xarray/blob/36f05d70c864ee7c61603c8a43ba721bf7f434b3/xarray/ufuncs.py#L47-L49

• [x] Closes #6016

cftime 1.5.2 renames "gregorian" to "standard". AFAIK this only changes the repr of cftime indices and does not seem to influence the creation of cftime indices.

In #6096 I realized that DuckArrayModule("dask") is called a lot in our tests - 145'835 times. Most of those are from is_dask_collection (is_duck_dask_array) This change avoids that the instance needs to be built every time.

```python import xarray as xr

%timeit xr.core.pycompat.DuckArrayModule("dask").available %timeit xr.core.pycompat.dsk.available ```

18.9 µs ± 97.7 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) 77.1 ns ± 1.22 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

Which leads to an incredible speed up of our tests of about 2.7 s :grin: ((18.9 - 0.0771) * 145835 / 1e6).

• [x] Closes #4601
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

It's not pretty as I had to define a number of empty methods... I think this should wait for 0.17

• [x] Closes #4727
• [ ] Tests added
• [ ] Passes isort . && black . && mypy . && flake8
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

This adds a dtype check for equal, identical, broadcast_equal, and the xr.testing.assert_* functions. It is far from complete: tests and documentation are still missing, but I wanted to get it online for feedback.

When I set check_dtype=True there are around 600 failures. Fixing that is for another PR. #4759 should help a bit.

• [ ] I added the checks to lazy_array_equiv, however, sometimes dask can get the dtype wrong before the compute (see below). Do you think I need to put it in the non-lazy part?

```python import numpy as np import xarray as xr

da = xr.DataArray(np.array([0, np.nan], dtype=object)).chunk()

da.prod().dtype # -> dtype('O') da.prod().compute().dtype # -> dtype('int64')

```

• [ ] check_dtype is still missing from assert_duckarray_allclose & assert_duckarray_equal - do you think there are required?

• [ ] The dtypes of array elements are not tested (see below). I don't think I'll implement that here.

```python da0 = xr.DataArray(np.array([0], dtype=object)) da1 = xr.DataArray(np.array([0.], dtype=object))

xr.testting.assert_equal(da0, da1, check_dtype=True) ```

• dask 2022.01 is out, so we can remove the pin

I hunted down some version checks that should no longer be necessary as we have moved beyond the minimum versions.

• [x] see #6186

I think it would be good to re-enable the upstream CI, even if this means we have to stick to py3.9 for the moment. I just subscribed to pydata/bottleneck#378, so I should see when we can switch to 3.10.

typing fixes for mypy 0.931 and numpy 1.22. Also tested with numpy 1.20 which probably many still have installed.

```python import xarray as xr import numpy as np

ds = xr.DataArray([np.NaN, np.NaN])

ds.mean() ds.sum()

```

Problem description

ds.mean() returns NaN, ds.sum() returns 0. This comes from numpy (cp np.nanmean vs. np.nansum), so it might have to be discussed upstream, but I wanted to ask the xarray community first on their opinion. This is also relevant for #422 (what happens if the all weights are NaN or sum up to 0).

Expected Output

I would expect both to return np.nan.

Output of xr.show_versions()

Our CI fails with some pytest-xdist error. Let's see if we get a clearer picture when disabling parallel tests. (Maybe some interaction between dask and pytest-xdist?).

• [x] Closes #6101

Reverts pydata/xarray#6077 (after dask has been pinned in #6111)

I disabled pytest-xdist because a dask issue renders our CI unusable. As soon as dask releases a new version we should revert #6077 again.

h5netcdf no longer warns for invalid netCDF (unless passing save_kwargs = {"invalid_netcdf": True}. We need to adapt our tests.

@kmuehlbauer

edit: I added h5netcdf to the upstream tests - I can also revert this change if you prefer.

• [x] Closes #5872
• [x] Passes pre-commit run --all-files

pandas will disallow skipna=None (pandas-dev/pandas#44178) - this fixes a test which relies on this. I don't think we have any user facing use of this. AFAIK we don't use pandas for reductions anywhere)

Fixes one part of #5872

pseudoNETCDF adds two attrs to ict files, which breaks the following two tests:

Test 1: https://github.com/pydata/xarray/blob/07de257c5884df49335496ee6347fb633a7c302c/xarray/tests/test_backends.py#L3944 Test 2:

https://github.com/pydata/xarray/blob/07de257c5884df49335496ee6347fb633a7c302c/xarray/tests/test_backends.py#L4030

I reproduced the test file so that the tests pass again. To reproduce the file I used the following bit of code:

```python import xarray as xr from xarray.tests import test_backends

fN = "xarray/tests/data/example.ict" fmtkw = {"format": "ffi1001"}

ds = xr.open_dataset(fN, engine="pseudonetcdf", backend_kwargs={"format": "ffi1001"})

c = test_backends.TestPseudoNetCDFFormat() c.save(ds, fN, **fmtkw) ```

The save method is here:

https://github.com/pydata/xarray/blob/07de257c5884df49335496ee6347fb633a7c302c/xarray/tests/test_backends.py#L4124

@barronh I would appreciate your review here - I am not sure if this is the right approach.

• [x] Part of #4513
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

This PR warns when passing keep_attrs to resample(..., keep_attrs=True) and rolling_exp(..., keep_attrs=True) as they have no effect (rightfully). Also removes keep_attrs from the docstring of resample.

• [x] Part of #4513 (maybe the last one - need to double check)
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

As per https://github.com/pydata/xarray/issues/3891#issuecomment-612522628 I also changed the default to True.

Another enormous PR from my side ;) unpin pip again. numpy probably fixed the issue re the name of the nightly build. But I also need to doublecheck if scipy is ok.

• [x] Closes #4858
• [x] Passes pre-commit run --all-files

iris moved its documentation to https://scitools-iris.readthedocs.io/en/stable/

There is a compatibility issue with h5py v3. Pin h5py to version 2 for the moment. I can open an issue shortly.

• [x] Closes #4521
• [x] Passes isort . && black . && mypy . && flake8

That was fast - eccodes already fixed the issue.

and put the comment on the same line

• [x] Closes #3756
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [x] Closes #2019
• [ ] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API
• [x] reverts #2021

Let's see if this passes....

• [x] Closes #3634
• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

np.nanquantile was added in numpy 1.15.0, the current minimum requirement for numpy is 1.14.0, therefore we have to test this ourselves.

• [x] Closes #5517
• [x] Passes pre-commit run --all-files

The minimal typing for numpy 1.21. As always I am by no means a typing specialist.

Code Sample, a copy-pastable example if possible

```python import numpy as np import xarray as xr

x = np.arange(1, 366) y = np.random.randn(365) ds = xr.DataArray(y, dims=dict(dayofyear=x))

ds.rolling(center=True, dayofyear=31).mean() ```

Problem description

rolling cannot directly handle periodic boundary conditions (lon, dayofyear, ...), but could be very helpful to e.g. calculate climate indices. Also I cannot really think of an easy way to append the first elements to the end of the dataset and then calculate rolling.

Is there a way to do this? Should xarray support this feature?

This might also belong to SO...

What happened:

combine_by_coords can succeed when it should not - depending on the name of the dimensions (which determines the order of operations in combine_by_coords).

What you expected to happen:

• I think it should throw an error in both cases.

Minimal Complete Verifiable Example:

```python import numpy as np import xarray as xr

data = np.arange(5).reshape(1, 5) x = np.arange(5) x_name = "lat"

da0 = xr.DataArray(data, dims=("t", x_name), coords={"t": [1], x_name: x}).to_dataset(name="a") x = x + 1e-6 da1 = xr.DataArray(data, dims=("t", x_name), coords={"t": [2], x_name: x}).to_dataset(name="a") ds = xr.combine_by_coords((da0, da1))

ds ```

returns: python <xarray.Dataset> Dimensions: (lat: 10, t: 2) Coordinates: * lat (lat) float64 0.0 1e-06 1.0 1.0 2.0 2.0 3.0 3.0 4.0 4.0 * t (t) int64 1 2 Data variables: a (t, lat) float64 0.0 nan 1.0 nan 2.0 nan ... 2.0 nan 3.0 nan 4.0 Thus lat is interlaced - it don't think combine_by_coords should do this. If you set

python x_name = "lat" and run the example again, it returns:

```python-traceback ValueError: Resulting object does not have monotonic global indexes along dimension x

```

Anything else we need to know?:

• this is vaguely related to #4077 but I think it is separate
• combine_by_coords concatenates over all dimensions where the coords are different - therefore compat="override" doesn't actually do anything? Or does it?

https://github.com/pydata/xarray/blob/ba42c08af9afbd9e79d47bda404bf4a92a7314a0/xarray/core/combine.py#L69

cc @dcherian @TomNicholas

Environment:

In #4622 @toddrjen points out that xr.testing.assert_equal does not test for the dtype, only for the value. Therefore the following does not raise an error:

```python import numpy as np import xarray as xr import pandas as pd

xr.testing.assert_equal( xr.DataArray(np.array(1, dtype=int)), xr.DataArray(np.array(1, dtype=float)) ) xr.testing.assert_equal( xr.DataArray(np.array(1, dtype=int)), xr.DataArray(np.array(1, dtype=object)) ) xr.testing.assert_equal( xr.DataArray(np.array("a", dtype=str)), xr.DataArray(np.array("a", dtype=object)) ) ```

This comes back to numpy, i.e. the following is True:

python np.array(1, dtype=int) == np.array(1, dtype=float)

Depending on the situation one or the other is desirable or not. Thus, I would suggest to add a check_dtype argument to xr.testing.assert_equal and also to DataArray.equals (and Dataset and Variable and identical). I have not seen such an option in numpy, but pandas has it (e.g. pd.testing.assert_series_equal(left, right, check_dtype=True, ...). I would not change __eq__.

• Thoughts?
• What should the default be? We could try True first and see how many failures this creates?
• What to do with coords and indexes? pd.testing.assert_series_equal has a check_index_type keyword. Probably we need check_coords_type as well? This makes the whole thing much more complicated... Also #4543

MCVE Code Sample

```python import numpy as np import xarray as xr import matplotlib as mpl import matplotlib.pyplot as plt

data = xr.DataArray(np.arange(24).reshape(4, 6))

data.plot.contour(vmax=10, add_colorbar=True) ```

Expected Output

python h = plt.contour(data.values, vmax=10) plt.colorbar(h)

Problem Description

A contour(vmax=vmax) plot differs between xarray and matplotlib. I think the problem is here:

https://github.com/pydata/xarray/blob/95e4f6c7a636878c94b892ee8d49866823d0748f/xarray/plot/utils.py#L265

xarray calculates the levels from vmax while matplotlib (probably) calculates the levels from data.max() and uses vmax only for the norm. For contourf and pcolormesh this is not so relevant as the capped values are then drawn with the over color. However, there may also be a good reason for this behavior.

Output of xr.show_versions()

There was a discussion on the performance of the weighted mean/ sum in terms of memory footprint but also speed, and there may indeed be some things that can be optimized. See the posts at the end of the PR. However, the optimal implementation will probably depend on the use case and some profiling will be required.

I'll just open an issue to keep track of this. @seth-p

• do not change user-specified units
• always format infered units as 'YYYY-mmmm-ddTHH:MM:SS'

This is my naïve approach.

• [ ] Closes #1449
• [ ] Tests added / passed
• [ ] Passes git diff upstream/master | flake8 --diff
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

Silences a number of warnings that accumulated in our test suite (c.f. #3266). The changes are mostly unrelated but small.

matplotlib warns: Calling gca() with keyword arguments was deprecated in Matplotlib 3.4. Starting two minor releases later, gca() will take no keyword arguments. The gca() function should only be used to get the current axes, or if no axes exist, create new axes with default keyword arguments. To create a new axes with non-default arguments, use plt.axes() or plt.subplot().

This only uses plt.gca() if there are active axes, else it calls plt.axes(**kwargs). Note that this can silently ignore some arguments. However, that this is already the case.