issues

What happened?

Yesterday we got a dependabot update PR to upgrade xarray from 2022.10.0 to 2022.11.0 and a test where we check for our own deprecation warnings failed because there was an additional unexpected warning. After some debugging we found that the warning was caused by calling xarray.Dataset.to_netcdf for the first time in our test suite, but did not trigger when calling it again.

After a lot of head-scratching and confusion, we found that it is an import order problem that can be solved by importing netCDF4 before importing xarray (we didn't import netCDF4 at all in our code).

What did you expect to happen?

No RuntimeWarning from netCDF4

Minimal Complete Verifiable Example

Python import xarray import warnings warnings.filterwarnings('error') import netCDF4

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

Python Traceback (most recent call last): File "d:\git\pyglotaran\glotaran\builtin\io\folder\test\test_folder_plugin.py", line 86, in <module> save_result(result_path="foo", format_name="folder", result=result) File "D:\git\pyglotaran\glotaran\plugin_system\io_plugin_utils.py", line 87, in wrapper return func(*args, **kwargs) File "D:\git\pyglotaran\glotaran\plugin_system\project_io_registration.py", line 473, in save_result paths = io.save_result( # type: ignore[call-arg] File "D:\git\pyglotaran\glotaran\builtin\io\folder\folder_plugin.py", line 192, in save_result save_dataset( File "D:\git\pyglotaran\glotaran\plugin_system\io_plugin_utils.py", line 87, in wrapper return func(*args, **kwargs) File "D:\git\pyglotaran\glotaran\plugin_system\data_io_registration.py", line 242, in save_dataset io.save_dataset( # type: ignore[call-arg] File "D:\git\pyglotaran\glotaran\builtin\io\netCDF\netCDF.py", line 24, in save_dataset data_to_save.to_netcdf(file_name, mode="w") File "C:\Anaconda3\envs\pyglotaran310\lib\site-packages\xarray\core\dataset.py", line 1903, in to_netcdf return to_netcdf( # type: ignore # mypy cannot resolve the overloads:( File "C:\Anaconda3\envs\pyglotaran310\lib\site-packages\xarray\backends\api.py", line 1176, in to_netcdf engine = _get_default_engine(path_or_file) File "C:\Anaconda3\envs\pyglotaran310\lib\site-packages\xarray\backends\api.py", line 140, in _get_default_engine return _get_default_engine_netcdf() File "C:\Anaconda3\envs\pyglotaran310\lib\site-packages\xarray\backends\api.py", line 118, in _get_default_engine_netcdf import netCDF4 # noqa: F401 File "C:\Anaconda3\envs\pyglotaran310\lib\site-packages\netCDF4\__init__.py", line 3, in <module> from ._netCDF4 import * File "src\netCDF4\_netCDF4.pyx", line 1, in init netCDF4._netCDF4 RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject

Anything else we need to know?

The problem can be reproduced by running console python -c "import xarray;import warnings;warnings.filterwarnings('error');import netCDF4"

which throws the error python Traceback (most recent call last): File "<string>", line 1, in <module> File "C:\Anaconda3\envs\xarray\lib\site-packages\netCDF4\__init__.py", line 3, in <module> from ._netCDF4 import * File "src\netCDF4\_netCDF4.pyx", line 1, in init netCDF4._netCDF4 RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject

When importing netCDF4 first all runs as expected console python -c "import netCDF4;import xarray;import warnings;warnings.filterwarnings('error');import netCDF4"

A git bisect shows that the first commit with this problem was https://github.com/pydata/xarray/commit/f32d354e295c05fb5c5ece7862f77f19d82d5894

Environment

Hi there, we just wanted to upgrade xarray to version 0.21.0 and our ASV workflow on the CI started failing.

python File "/home/runner/work/pyglotaran/pyglotaran/benchmark/.asv/env/76391772e92136ec87b9940d70226329/lib/python3.8/site-packages/xarray/core/dask_array_compat.py", line 4, in <module> from packaging.version import Version ModuleNotFoundError: No module named 'packaging' The reason is that packaging isn't a direct dependency of xarray.

```console $ johnnydep xarray 2022-01-29 08:28:38 [info ] init johnnydist [johnnydep.lib] dist=xarray parent=None 2022-01-29 08:28:40 [info ] init johnnydist [johnnydep.lib] dist=numpy>=1.18 parent=xarray 2022-01-29 08:28:44 [info ] init johnnydist [johnnydep.lib] dist=pandas>=1.1 parent=xarray 2022-01-29 08:28:47 [info ] init johnnydist [johnnydep.lib] dist=numpy>=1.18.5 parent=pandas>=1.1 2022-01-29 08:28:51 [info ] init johnnydist [johnnydep.lib] dist=python-dateutil>=2.8.1 parent=pandas>=1.1 2022-01-29 08:28:53 [info ] init johnnydist [johnnydep.lib] dist=pytz>=2020.1 parent=pandas>=1.1 2022-01-29 08:28:56 [info ] init johnnydist [johnnydep.lib] dist=six>=1.5 parent=python-dateutil>=2.8.1 name summary

xarray N-D labeled arrays and datasets in Python ├── numpy>=1.18 NumPy is the fundamental package for array computing with Python. └── pandas>=1.1 Powerful data structures for data analysis, time series, and statistics ├── numpy>=1.18.5 NumPy is the fundamental package for array computing with Python. ├── python-dateutil>=2.8.1 Extensions to the standard Python datetime module │ └── six>=1.5 Python 2 and 3 compatibility utilities └── pytz>=2020.1 World timezone definitions, modern and historical ```

The problem why your tests didn't catch this is that packaging is a dependency of pytest itself. ```console $ johnnydep pytest 2022-01-29 08:25:03 [info ] init johnnydist [johnnydep.lib] dist=pytest parent=None 2022-01-29 08:25:07 [info ] init johnnydist [johnnydep.lib] dist=atomicwrites>=1.0 parent=pytest 2022-01-29 08:25:09 [info ] init johnnydist [johnnydep.lib] dist=attrs>=19.2.0 parent=pytest 2022-01-29 08:25:11 [info ] init johnnydist [johnnydep.lib] dist=colorama parent=pytest 2022-01-29 08:25:14 [info ] init johnnydist [johnnydep.lib] dist=iniconfig parent=pytest 2022-01-29 08:25:16 [info ] init johnnydist [johnnydep.lib] dist=packaging parent=pytest 2022-01-29 08:25:18 [info ] init johnnydist [johnnydep.lib] dist=pluggy<2.0,>=0.12 parent=pytest 2022-01-29 08:25:20 [info ] init johnnydist [johnnydep.lib] dist=py>=1.8.2 parent=pytest 2022-01-29 08:25:23 [info ] init johnnydist [johnnydep.lib] dist=toml parent=pytest 2022-01-29 08:25:25 [info ] init johnnydist [johnnydep.lib] dist=pyparsing!=3.0.5,>=2.0.2 parent=packaging name summary

pytest pytest: simple powerful testing with Python ├── atomicwrites>=1.0 Atomic file writes. ├── attrs>=19.2.0 Classes Without Boilerplate ├── colorama Cross-platform colored terminal text. ├── iniconfig iniconfig: brain-dead simple config-ini parsing ├── packaging Core utilities for Python packages │ └── pyparsing!=3.0.5,>=2.0.2 Python parsing module ├── pluggy<2.0,>=0.12 plugin and hook calling mechanisms for python ├── py>=1.8.2 library with cross-python path, ini-parsing, io, code, log facilities └── toml Python Library for Tom's Obvious, Minimal Language ```

We also only saw our ASV workflow fail since ASV only installs the package and all direct runtime dependencies in the venv it runs the benchmark with.

I prepared a dummy PR on my fork to demonstrate that this fixes the issue.

Since the release is only 10h old maybe just do a quick post-release so most users won't even notice.

• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

Affected xarray versions 0.20.0+

When dependabot recently made a PR to update xarray to 0.20.0 our integration test CI failed with .... File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/xarray/plot/plot.py", line 447, in line ax.set_title(darray._title_for_slice()) File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/xarray/core/dataarray.py", line 3140, in _title_for_slice v=format_item(coord.values), File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/xarray/core/formatting.py", line 146, in format_item return f"{x:.4}" TypeError: unsupported format string passed to numpy.ndarray.__format__

We use xarray.Dataarray.plot.line(x="<dim_name>") to plot data of shape (N, M) where N and M are >=1.

The crash is caused because the following line

https://github.com/pydata/xarray/blob/402a9b398868bd9bd385210138d6f8ea610c8c40/xarray/core/dataarray.py#L3136

allows arrays of shape () and (1,) to be passed into format_item, where both pass this check

https://github.com/pydata/xarray/blob/402a9b398868bd9bd385210138d6f8ea610c8c40/xarray/core/formatting.py#L145-L146

But f"{x:.4}" only works for arrays of shape () and crashes on arrays of shape (1,)

Before the check was https://github.com/pydata/xarray/blob/a78c1e0115d38cb4461fd1aba93334d440cff49c/xarray/core/formatting.py#L145-L146

Which didn't allow any arrays and caused the float formatting issue.

I hope TestPlot1D was the correct place to put the test since the plot itself is 1D but the data which are plotted have a second dimension with size 1 (naming thing and where to put things... 😅).

• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

PS: Thanks for the awesome package ❤️