issues: 479190812
This data as json
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | active_lock_reason | draft | pull_request | body | reactions | performed_via_github_app | state_reason | repo | type |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
479190812 | MDU6SXNzdWU0NzkxOTA4MTI= | 3200 | open_mfdataset memory leak, very simple case. v0.12 | 19933988 | open | 0 | 7 | 2019-08-09T22:38:39Z | 2023-02-03T22:58:32Z | NONE | MCVE Code Sample```python import glob import xarray as xr import numpy as np from memory_profiler import profile def CreateTestFiles(): # create a bunch of files xlen = int(1e2) ylen = int(1e2) xdim = np.arange(xlen) ydim = np.arange(ylen)
@profile def ReadFiles(): xr.open_mfdataset(glob.glob('testfiles/*'), concat_dim='time') if name == 'main': # write out files for testing CreateTestFiles()
~ ~ ``` usage:mprof run simplest_case.py mprof plot (mprof is a python memory profiling library) Problem Descriptiondask version 1.1.4 xarray version 0.12 python 3.7.3 There appears to be a persistent memory leak in open_mfdataset. I'm creating a model calibration script that runs for ~1000 iterations, opening and closing the same set of files (dimensions are the same, but the data is different) with each iteration. I eventually run out of memory because of the leak. This simple case captures the same behavior. Closing the files with .close() does not fix the problem. Is there a work around for this? I've perused some of the issues but cannot tell if this has been resolved. Output of
|
{ "url": "https://api.github.com/repos/pydata/xarray/issues/3200/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
13221727 | issue |