issues: 1512460818
This data as json
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | active_lock_reason | draft | pull_request | body | reactions | performed_via_github_app | state_reason | repo | type |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1512460818 | I_kwDOAMm_X85aJlIS | 7404 | Memory leak - xr.open_dataset() not releasing memory. | 60647051 | open | 0 | 7 | 2022-12-28T06:40:03Z | 2023-06-23T18:54:34Z | NONE | What happened?Let's take this sample netcdf file. Observe that the memory has not been cleared even after deleting the ds. Code ```bash import os import psutil import xarray as xr from memory_profiler import profile @profile def main(): path = 'ECMWF_ERA-40_subset.nc' print(f"Before opening file: {psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2} MiB") ds = xr.open_dataset(path) del ds print(f"After opening file: {psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2} MiB") if name == 'main': print(f"Start: {psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2} MiB") main() print(f"End: {psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2} MiB") ``` Console logs ```bash Start: 186.5859375 MiB Before opening file: 187.25 MiB After opening file: 308.09375 MiB Filename: temp.py Line # Mem usage Increment Occurrences Line Contents
End: 308.09375 MiB ``` I am using xarray==0.20.2and gdal==3.5.1. Sister issue: https://github.com/ecmwf/cfgrib/issues/325#issuecomment-1363011917 What did you expect to happen?Ideally, memory consumed by the xarray dataset should be released when the dataset is closed/deleted. Minimal Complete Verifiable ExampleNo response MVCE confirmation
Relevant log outputNo response Anything else we need to know?No response Environment
INSTALLED VERSIONS
------------------
commit: None
python: 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:53)
[GCC 9.4.0]
python-bits: 64
OS: Linux
OS-release: 4.19.0-22-cloud-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: C.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.12.2
libnetcdf: 4.8.1
xarray: 0.20.2
pandas: 1.3.5
numpy: 1.19.5
scipy: 1.7.3
netCDF4: 1.6.0
pydap: None
h5netcdf: 1.0.2
h5py: 3.7.0
Nio: None
zarr: 2.12.0
cftime: 1.6.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.2.10
cfgrib: 0.9.10.1
iris: None
bottleneck: None
dask: 2022.02.0
distributed: 2022.02.0
matplotlib: 3.5.2
cartopy: 0.20.3
seaborn: 0.11.2
numbagg: None
fsspec: 2022.7.1
cupy: None
pint: None
sparse: None
setuptools: 59.8.0
pip: 22.2.2
conda: 22.9.0
pytest: None
IPython: 7.33.0
sphinx: None
|
{ "url": "https://api.github.com/repos/pydata/xarray/issues/7404/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
13221727 | issue |