What is your issue?

Hi team — apologies but I mistakenly pushed to main. Less than a minute later I pushed another commit reverting the commit.

I'll check my git shortcuts tomorrow to ensure this can't happen by default — I thought I had the push remote set correctly, but possibly something doesn't use that.

Would we consider adding protection from pushes to main? (Am not blaming that setting tbc...)

(Will close this issue)

The typing of Variable.data has been the very wide Variable.data: Any because of the difficulties defining the typing of a duckarray.

This has led to confusion in downstream functions that uses .data. Sometimes arguments uses Any, np.typing.ArrayLike, np.ndarray or dask array typing which is unnecessarily narrow when xarray also supports for example sparse arrays or cupy arrays.

This PR is a start at cleaning this up by defining a T_DuckArray typevar that symbolizes an array api compliant array.

What is your issue?

We have recently had a few reports expressing frustration with our minimum dependency version policy. This issue aims to discuss if changes to our policy are needed.

Background

1. Our current minimum dependency versions policy reads:

   Minimum dependency versions

   Xarray adopts a rolling policy regarding the minimum supported version of its dependencies:

   • Python: 24 months (NEP-29)
   • numpy: 18 months (NEP-29)
   • all other libraries: 12 months

   This means the latest minor (X.Y) version from N months prior. Patch versions (x.y.Z) are not pinned, and only the latest available at the moment of publishing the xarray release is guaranteed to work.

   You can see the actual minimum tested versions:

   pydata/xarray

2. We have a script that checks versions and dates and advises us on when to bump minimum versions.

   https://github.com/pydata/xarray/blob/main/ci/min_deps_check.py

Diagnosis

1. Our policy and min_deps_check.py script have greatly reduced our deliberations on which versions to support and the maintenance burden of supporting out dated versions of dependencies.
2. We likely need to update our policy and min_deps_check.py script to properly account for Python's SEMVER bugfix releases. Depending on how you interpret the policy, we may have prematurely dropped Python 3.8 (see below for a potential action item).

Discussion questions

1. Is the policy working as designed, are the support windows documented above still appropriate for where Xarray is at today?
2. Is this policy still in line with how our peer libraries are operating?

Action items

1. There is likely a bug in the patch-version comparison in the minimum Python version. Moreover, we don't differentiate between bugfix and security releases. I suggest we have a special policy for our minimum supported Python version that reads something like: > Python: 24 months from the last bugfix release (security releases are not considered).

xref: https://github.com/pydata/xarray/issues/4179, https://github.com/pydata/xarray/pull/7461

Moderators note: I suspect a number of folks will want to comment on this issue with "Please support Python 3.8 for longer...". If that is the nature of your comment, please just give this a ❤️ reaction rather than filling up the discussion.

• [x] Closes #8090
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• Complements #8102
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

I don't think that we need to copy most API from Dataset / DataArray to Coordinates, but I find it convenient to have some relevant methods there too. For example, building Coordinates from scratch (with custom indexes) before passing the whole coords + indexes bundle around:

```python import dask.array as da import numpy as np import xarray as xr

coords = ( xr.Coordinates( coords={"x": da.arange(100_000_000), "y": np.arange(100)}, indexes={}, ) .set_xindex("x", DaskIndex) .set_xindex("y", xr.indexes.PandasIndex) )

ds = xr.Dataset(coords=coords)

<xarray.Dataset>

Dimensions: (x: 100000000, y: 100)

Coordinates:

* x (x) int64 dask.array<chunksize=(16777216,), meta=np.ndarray>

* y (y) int64 0 1 2 3 4 5 6 7 8 9 10 ... 90 91 92 93 94 95 96 97 98 99

Data variables:

empty

Indexes:

x DaskIndex

```

Documents a hidden feature I noticed yesterday, corrects incorrect docstrings, and tidies up some of the typing internally.

• [ ] ~~Closes #xxxx~~
• [ ] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] ~~New functions/methods are listed in api.rst~~

• [x] Fixes #6870
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

A similar change should solve #6528, but let's get one finished first...

~Posting for discussion, will comment inline~ Ready for merge

What happened?

We lose the time coord here — Dimensions without coordinates: time:

```python ds = xr.tutorial.load_dataset("air_temperature") ds.rolling_exp(time=5).mean()

<xarray.Dataset> Dimensions: (lat: 25, time: 2920, lon: 53) Coordinates: * lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0 * lon (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0 Dimensions without coordinates: time Data variables: air (time, lat, lon) float32 241.2 242.5 243.5 ... 296.4 296.1 295.7 ```

(I realize I wrote this, I didn't think this used to happen, but either it always did or I didn't write good enough tests... mea culpa)

What did you expect to happen?

We keep the time coords, like we do for normal rolling:

python In [2]: ds.rolling(time=5).mean() Out[2]: <xarray.Dataset> Dimensions: (lat: 25, lon: 53, time: 2920) Coordinates: * lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0 * lon (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0 * time (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00

Minimal Complete Verifiable Example

Python (as above)

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

What is your issue?

The h5netcdf engine for reading NetCDF4-files was recently changed https://github.com/h5netcdf/h5netcdf/pull/151 so that when reading attributes, any 1D array/list of length 1 gets turned into a scalar element/item. The change happened with version 0.14.0.

The issue is that the xarray documentation still describes the old h5netcdf-behaviour on https://docs.xarray.dev/en/stable/user-guide/io.html?highlight=attributes%20h5netcdf#netcdf

Could we mention this also on https://docs.xarray.dev/en/stable/generated/xarray.open_dataset.html#xarray.open_dataset under the engine argument, or just make sure it links to the above page?

I initially looked under https://docs.xarray.dev/en/stable/user-guide/io.html?highlight=string#string-encoding because my issue was for a string array/list, but maybe too much to mention there if this is a general change that affects attributes of all types.

As explained on the h5netcdf-issue tracker, the reason for dropping/squeezing 1-length-array attributes to scalars, is for compatibility with the other NetCDF4-engine or NetCDF in general (and there might be some varying opinions about how good that is, vs. fully using features available in HDF5). (Interesting to note is that when writing, an attribute with a python list of length 1 does give an array of length 1 in the HDF5/NetCDF4-file, the dropping of array dimension only happens only when reading.)

Adding the invalid_netcdf=True argument when loading does not change the behaviour. Maybe it could be interesting to use it to generally allow 1-length attribute arrays? Now, I think every usage of array-attributes will need conversions like list_in_recent_version = attribute if isinstance(attribute, list) else [attribute] or always_list = list(attribute if isinstance(attribute, (list, np.ndarray)) else [attribute]) to support both old and new versions. Otherwise, iterating over an attribute string will cause surprises by iterating over its characters instead of doing a single iteration that yields the single string (as in older versions).

Minimal example

This serves to clarify what happens. The issue is not about reverting to the old behaviour (although I liked it), just updating the xarray documentation. ``` import xarray as xr import numpy as np ds = xr.Dataset() ds['stuff'] = xr.DataArray(np.random.randn(2), dims='x') ds['stuff'].attrs['strings_0D_one'] = 'abc' ds['stuff'].attrs['strings_1D_two'] = ['abc', 'def'] ds['stuff'].attrs['strings_1D_one'] = ['abc'] path = 'demo.nc' ds.to_netcdf(path, engine='h5netcdf', format='netCDF4') ds2 = xr.load_dataset(path, engine='h5netcdf')

print(type(ds2['stuff'].attrs['strings_0D_one']).name, repr(ds2['stuff'].attrs['strings_0D_one'])) print(type(ds2['stuff'].attrs['strings_1D_two']).name, repr(ds2['stuff'].attrs['strings_1D_two'])) print(type(ds2['stuff'].attrs['strings_1D_one']).name, repr(ds2['stuff'].attrs['strings_1D_one'])) ``` With h5netcdf: 0.12.0 (python: 3.7.9, OS: Windows, OS-release: 10, libhdf5: 1.10.4, xarray: 0.20.1, pandas: 1.3.4, numpy: 1.21.5, netCDF4: None, h5netcdf: 0.12.0, h5py: 2.10.0) the printouts are:

str 'abc' ndarray array(['abc', 'def'], dtype=object) ndarray array(['abc'], dtype=object)

With h5netcdf: 1.0.0 (python: 3.8.11, OS: Linux, OS-release: 3.10.0-1160.49.1.el7.x86_64, libhdf5: 1.10.4, xarray: 0.20.1, pandas: 1.4.2, numpy: 1.21.2, netCDF4: None, h5netcdf: 1.0.0, h5py: 2.10.0) the printouts are:

str 'abc' list ['abc', 'def'] str 'abc'

I have tested that direct reading by h5py.File gives str, ndarray, ndarray so the change is not in the writing or h5py.

• [x] Closes #6570