issues: 1323734180

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
1323734180 I_kwDOAMm_X85O5pSk 6854 test_open_nczarr uses too much memory 302469 closed 0     8 2022-08-01T03:02:27Z 2023-05-08T09:42:09Z 2023-05-08T09:42:08Z CONTRIBUTOR      

What happened?

I'm updating builds for Fedora to 2022.06.0, and running tests, the process runs out of memory and is OOM-killed. Running all the steps manually, this gets to the to_netcdf call in createTestNCZarr._create_nczarr. At that point, memory usage rises to 8.9G resident / 17.7G virtual, and the process is killed (actually, the entire SSH session is killed).

What did you expect to happen?

Tests pass without issue.

Minimal Complete Verifiable Example

Python import xarray as xr from xarray.tests.test_dataset import create_test_data ds = create_test_data() ds = ds.drop_vars("dim3") ds.to_netcdf(f"file://foo.zarr#mode=nczarr,noxarray")

MVCE confirmation

  • [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • [X] Complete example — the example is self-contained, including all data and the text of any traceback.
  • [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

Python Jul 31 22:56:07 hostnmme systemd-oomd[805]: Killed /user.slice/user-1000.slice/session-131.scope due to memory used (15294935040) / total (15569043456) and swap used (15308353536) / total (16978534400) being more than 90.00% Jul 31 22:56:07 hostname systemd[1]: session-131.scope: systemd-oomd killed 6 process(es) in this unit.

Anything else we need to know?

No response

Environment

INSTALLED VERSIONS ------------------ commit: None python: 3.11.0b5 (main, Jul 26 2022, 00:00:00) [GCC 12.1.1 20220628 (Red Hat 12.1.1-3)] python-bits: 64 OS: Linux OS-release: 5.17.13-300.fc36.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: C.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.12.1 libnetcdf: 4.9.0 xarray: 2022.6.0 pandas: 1.3.5 numpy: 1.22.0 scipy: 1.8.1 netCDF4: 1.6.0 pydap: None h5netcdf: None h5py: None Nio: None zarr: 2.11.3 cftime: 1.6.0 nc_time_axis: None PseudoNetCDF: None rasterio: 1.2.10 cfgrib: None iris: None bottleneck: None dask: 2022.7.1 distributed: None matplotlib: 3.5.2 cartopy: None seaborn: 0.11.2 numbagg: None fsspec: 2022.5.0 cupy: None pint: 0.16.1 sparse: None flox: None numpy_groupies: None setuptools: 62.6.0 pip: 22.2 conda: None pytest: 7.1.2 IPython: None sphinx: 5.0.2 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/6854/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed 13221727 issue

Links from other tables