id,node_id,number,title,user,state,locked,assignee,milestone,comments,created_at,updated_at,closed_at,author_association,active_lock_reason,draft,pull_request,body,reactions,performed_via_github_app,state_reason,repo,type 1512460818,I_kwDOAMm_X85aJlIS,7404,Memory leak - xr.open_dataset() not releasing memory.,60647051,open,0,,,7,2022-12-28T06:40:03Z,2023-06-23T18:54:34Z,,NONE,,,,"### What happened? Let's take [this](https://www.unidata.ucar.edu/software/netcdf/examples/ECMWF_ERA-40_subset.nc) sample netcdf file. Observe that the memory has not been cleared even after deleting the ds. **Code** ```bash import os import psutil import xarray as xr from memory_profiler import profile @profile def main(): path = 'ECMWF_ERA-40_subset.nc' print(f""Before opening file: {psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2} MiB"") ds = xr.open_dataset(path) del ds print(f""After opening file: {psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2} MiB"") if __name__ == '__main__': print(f""Start: {psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2} MiB"") main() print(f""End: {psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2} MiB"") ``` **Console logs** ```bash Start: 186.5859375 MiB Before opening file: 187.25 MiB After opening file: 308.09375 MiB Filename: temp.py Line # Mem usage Increment Occurrences Line Contents ============================================================= 6 187.2 MiB 187.2 MiB 1 @profile 7 def main(): 8 187.2 MiB 0.0 MiB 1 path = 'ECMWF_ERA-40_subset.nc' 9 187.2 MiB 0.0 MiB 1 print(f""Before opening file: {psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2} MiB"") 10 308.1 MiB 120.8 MiB 1 ds = xr.open_dataset(path) 11 308.1 MiB 0.0 MiB 1 del ds 12 308.1 MiB 0.0 MiB 1 print(f""After opening file: {psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2} MiB"") End: 308.09375 MiB ``` I am using xarray==0.20.2and gdal==3.5.1. Sister issue: https://github.com/ecmwf/cfgrib/issues/325#issuecomment-1363011917 ### What did you expect to happen? Ideally, memory consumed by the xarray dataset should be released when the dataset is closed/deleted. ### Minimal Complete Verifiable Example _No response_ ### MVCE confirmation - [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray. - [ ] Complete example — the example is self-contained, including all data and the text of any traceback. - [ ] Verifiable example — the example copy & pastes into an IPython prompt or [Binder notebook](https://mybinder.org/v2/gh/pydata/xarray/main?urlpath=lab/tree/doc/examples/blank_template.ipynb), returning the result. - [ ] New issue — a search of GitHub Issues suggests this is not a duplicate. ### Relevant log output _No response_ ### Anything else we need to know? _No response_ ### Environment
INSTALLED VERSIONS ------------------ commit: None python: 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:53) [GCC 9.4.0] python-bits: 64 OS: Linux OS-release: 4.19.0-22-cloud-amd64 machine: x86_64 processor: byteorder: little LC_ALL: None LANG: C.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.12.2 libnetcdf: 4.8.1 xarray: 0.20.2 pandas: 1.3.5 numpy: 1.19.5 scipy: 1.7.3 netCDF4: 1.6.0 pydap: None h5netcdf: 1.0.2 h5py: 3.7.0 Nio: None zarr: 2.12.0 cftime: 1.6.1 nc_time_axis: None PseudoNetCDF: None rasterio: 1.2.10 cfgrib: 0.9.10.1 iris: None bottleneck: None dask: 2022.02.0 distributed: 2022.02.0 matplotlib: 3.5.2 cartopy: 0.20.3 seaborn: 0.11.2 numbagg: None fsspec: 2022.7.1 cupy: None pint: None sparse: None setuptools: 59.8.0 pip: 22.2.2 conda: 22.9.0 pytest: None IPython: 7.33.0 sphinx: None
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7404/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,issue