issues: 1437481995

What happened?

Yesterday we got a dependabot update PR to upgrade xarray from 2022.10.0 to 2022.11.0 and a test where we check for our own deprecation warnings failed because there was an additional unexpected warning. After some debugging we found that the warning was caused by calling xarray.Dataset.to_netcdf for the first time in our test suite, but did not trigger when calling it again.

After a lot of head-scratching and confusion, we found that it is an import order problem that can be solved by importing netCDF4 before importing xarray (we didn't import netCDF4 at all in our code).

What did you expect to happen?

No RuntimeWarning from netCDF4

Minimal Complete Verifiable Example

Python import xarray import warnings warnings.filterwarnings('error') import netCDF4

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

Python Traceback (most recent call last): File "d:\git\pyglotaran\glotaran\builtin\io\folder\test\test_folder_plugin.py", line 86, in <module> save_result(result_path="foo", format_name="folder", result=result) File "D:\git\pyglotaran\glotaran\plugin_system\io_plugin_utils.py", line 87, in wrapper return func(*args, **kwargs) File "D:\git\pyglotaran\glotaran\plugin_system\project_io_registration.py", line 473, in save_result paths = io.save_result( # type: ignore[call-arg] File "D:\git\pyglotaran\glotaran\builtin\io\folder\folder_plugin.py", line 192, in save_result save_dataset( File "D:\git\pyglotaran\glotaran\plugin_system\io_plugin_utils.py", line 87, in wrapper return func(*args, **kwargs) File "D:\git\pyglotaran\glotaran\plugin_system\data_io_registration.py", line 242, in save_dataset io.save_dataset( # type: ignore[call-arg] File "D:\git\pyglotaran\glotaran\builtin\io\netCDF\netCDF.py", line 24, in save_dataset data_to_save.to_netcdf(file_name, mode="w") File "C:\Anaconda3\envs\pyglotaran310\lib\site-packages\xarray\core\dataset.py", line 1903, in to_netcdf return to_netcdf( # type: ignore # mypy cannot resolve the overloads:( File "C:\Anaconda3\envs\pyglotaran310\lib\site-packages\xarray\backends\api.py", line 1176, in to_netcdf engine = _get_default_engine(path_or_file) File "C:\Anaconda3\envs\pyglotaran310\lib\site-packages\xarray\backends\api.py", line 140, in _get_default_engine return _get_default_engine_netcdf() File "C:\Anaconda3\envs\pyglotaran310\lib\site-packages\xarray\backends\api.py", line 118, in _get_default_engine_netcdf import netCDF4 # noqa: F401 File "C:\Anaconda3\envs\pyglotaran310\lib\site-packages\netCDF4\__init__.py", line 3, in <module> from ._netCDF4 import * File "src\netCDF4\_netCDF4.pyx", line 1, in init netCDF4._netCDF4 RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject

Anything else we need to know?

The problem can be reproduced by running console python -c "import xarray;import warnings;warnings.filterwarnings('error');import netCDF4"

which throws the error python Traceback (most recent call last): File "<string>", line 1, in <module> File "C:\Anaconda3\envs\xarray\lib\site-packages\netCDF4\__init__.py", line 3, in <module> from ._netCDF4 import * File "src\netCDF4\_netCDF4.pyx", line 1, in init netCDF4._netCDF4 RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject

When importing netCDF4 first all runs as expected console python -c "import netCDF4;import xarray;import warnings;warnings.filterwarnings('error');import netCDF4"

A git bisect shows that the first commit with this problem was https://github.com/pydata/xarray/commit/f32d354e295c05fb5c5ece7862f77f19d82d5894

```console $ git bisect start status: waiting for both good and bad commits (xarray) /d/git/xarray (main|BISECTING) $ git bisect good v2022.10.0 status: waiting for bad commit, 1 good commit known (xarray) /d/git/xarray (main|BISECTING) $ git bisect bad v2022.11.0 Bisecting: 23 revisions left to test after this (roughly 5 steps) [4944b9eb1483c1fbd0e86fd12f3fb894b325fb8d] Fix binning when labels are provided. (#7205) (xarray) /d/git/xarray ((4944b9eb...)|BISECTING) $ git bisect run python -c "import xarray;import warnings;warnings.filterwarnings('error');import netCDF4" running 'python' '-c' 'import xarray;import warnings;warnings.filterwarnings('\''error'\'');import netCDF4' Bisecting: 11 revisions left to test after this (roughly 4 steps) [f32d354e295c05fb5c5ece7862f77f19d82d5894] Lazy Imports (#7179) running 'python' '-c' 'import xarray;import warnings;warnings.filterwarnings('\''error'\'');import netCDF4' Traceback (most recent call last): File "<string>", line 1, in <module> File "C:\Anaconda3\envs\xarray\lib\site-packages\netCDF4\__init__.py", line 3, in <module> from ._netCDF4 import * File "src\netCDF4\_netCDF4.pyx", line 1, in init netCDF4._netCDF4 RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject Bisecting: 5 revisions left to test after this (roughly 3 steps) [b9aedd0155548ed0f34506ecc255b1688f07ffaa] set_coords docs: see also Dataset.assign_coords (#7230) running 'python' '-c' 'import xarray;import warnings;warnings.filterwarnings('\''error'\'');import netCDF4' Bisecting: 2 revisions left to test after this (roughly 2 steps) [65bfa4d10a529f00a9f9b145d1cea402bdae83d0] Actually make the fast code path return early for Aligner.align (#7222) running 'python' '-c' 'import xarray;import warnings;warnings.filterwarnings('\''error'\'');import netCDF4' Bisecting: 0 revisions left to test after this (roughly 1 step) [fc9026b59d38146a21769cc2d3026a12d58af059] Avoid loading any data for reprs (#7203) running 'python' '-c' 'import xarray;import warnings;warnings.filterwarnings('\''error'\'');import netCDF4' f32d354e295c05fb5c5ece7862f77f19d82d5894 is the first bad commit commit f32d354e295c05fb5c5ece7862f77f19d82d5894 Author: Mick <mick.niklas@gmail.com> Date: Fri Oct 28 18:25:39 2022 +0200 Lazy Imports (#7179) * fix typing of BackendEntrypoint * make backends lazy * make matplotlib lazy and add tests for lazy modules * make flox lazy * fix generated docs on windows... * try fixing test * make pycompat lazy * make dask.array lazy * add import xarray without numpy or pandas benchmark * improve error reporting in test * fix import benchmark * add lazy import to whats-new * fix lazy import test * fix typos * fix windows stuff again asv_bench/benchmarks/import.py | 12 +- doc/whats-new.rst | 2 + xarray/backends/cfgrib_.py | 27 ++-- xarray/backends/common.py | 15 ++- xarray/backends/h5netcdf_.py | 19 ++- xarray/backends/netCDF4_.py | 16 +-- xarray/backends/pseudonetcdf_.py | 13 +- xarray/backends/pydap_.py | 24 ++-- xarray/backends/pynio_.py | 13 +- xarray/backends/scipy_.py | 12 +- xarray/backends/zarr.py | 15 +-- xarray/convert.py | 3 +- xarray/core/_aggregations.py | 247 ++++++++++++++++++++++++++++------- xarray/core/dataset.py | 3 +- xarray/core/duck_array_ops.py | 31 +++-- xarray/core/formatting.py | 36 ++--- xarray/core/indexing.py | 6 +- xarray/core/missing.py | 4 +- xarray/core/parallel.py | 20 +-- xarray/core/pycompat.py | 20 ++- xarray/core/utils.py | 19 +++ xarray/core/variable.py | 15 +-- xarray/plot/utils.py | 9 +- xarray/tests/test_backends.py | 4 +- xarray/tests/test_computation.py | 4 +- xarray/tests/test_dask.py | 3 +- xarray/tests/test_dataset.py | 4 +- xarray/tests/test_duck_array_ops.py | 4 +- xarray/tests/test_missing.py | 4 +- xarray/tests/test_plugins.py | 61 ++++++++- xarray/tests/test_sparse.py | 4 +- xarray/tests/test_variable.py | 4 +- xarray/util/generate_aggregations.py | 13 +- 33 files changed, 445 insertions(+), 241 deletions(-) bisect found first bad commit ```

Environment

INSTALLED VERSIONS ------------------ commit: None python: 3.10.6 | packaged by conda-forge | (main, Oct 7 2022, 20:14:50) [MSC v.1916 64 bit (AMD64)] python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: AMD64 Family 23 Model 8 Stepping 2, AuthenticAMD byteorder: little LC_ALL: None LANG: de_DE.UTF-8 LOCALE: ('de_DE', 'cp1252') libhdf5: 1.12.1 libnetcdf: 4.8.1 xarray: 2022.11.0 pandas: 1.5.1 numpy: 1.23.4 scipy: 1.9.3 netCDF4: 1.6.1 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.6.2 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: None distributed: None matplotlib: 3.6.1 cartopy: None seaborn: None numbagg: None fsspec: None cupy: None pint: None sparse: None flox: None numpy_groupies: None setuptools: 59.8.0 pip: 22.2.2 conda: None pytest: 7.1.3 IPython: 8.5.0 sphinx: 5.2.3 None