id,node_id,number,title,user,state,locked,assignee,milestone,comments,created_at,updated_at,closed_at,author_association,active_lock_reason,draft,pull_request,body,reactions,performed_via_github_app,state_reason,repo,type 878481461,MDU6SXNzdWU4Nzg0ODE0NjE=,5276,open_mfdataset: Not a valid ID,11815787,closed,0,,,4,2021-05-07T05:34:02Z,2022-04-09T15:49:50Z,2022-04-09T15:49:50Z,NONE,,,,"I have about 601 NETCDF4 files saved using xarray. We try to use open_mfdataset to access these files. The main code calls this function many times. At the first few calls, it works fine, after for a while it throw the following error message ""RuntimeError: NetCDF: Not a valid ID"" ```python def func(xpath, spec): doc = deepcopy(spec) with xr.open_mfdataset(xpath + ""/*.nc"", concat_dim='maturity') as data: var_name= list(data.data_vars)[0] ar = data[var_name] maturity = spec['maturity'] ann = ar.cumsum(dim='maturity') ann = ann - 1 ar1 = ann.sel(maturity=maturity) doc['data'] = ar1.load().values return doc ``` **Environment**:
Output of xr.show_versions() INSTALLED VERSIONS ------------------ commit: None python: 3.6.8.final.0 python-bits: 64 OS: Linux OS-release: 5.4.0-1047-aws machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: C.UTF-8 LOCALE: en_US.UTF-8 xarray: 0.11.0 pandas: 0.24.1 numpy: 1.15.4 scipy: 1.2.0 netCDF4: 1.4.2 h5netcdf: None h5py: 2.9.0 Nio: None zarr: None cftime: 1.0.3.4 PseudonetCDF: None rasterio: None iris: None bottleneck: 1.2.1 cyordereddict: None dask: 1.1.1 distributed: 1.25.3 matplotlib: 3.0.2 cartopy: None seaborn: 0.9.0 setuptools: 40.7.3 pip: 19.0.1 conda: None pytest: 4.2.0 IPython: 7.1.1 sphinx: 1.8.4
This error also happens with xarray version 0.10.9 **Error trace**:
```python 2021-05-05 09:28:19,911, DEBUG 7621, sim_io.py:483 - load_unique_document(), xpa th=/home/ubuntu/runs/20210331_001/nominal_dfs/uk 2021-05-05 09:28:42,774, ERROR 7621, run_gov_ret.py:33 - (), Unknown error=NetCDF: Not a valid ID Traceback (most recent call last): File ""/home/ubuntu/dev/py36/python/ev/model/api3/run_gov_ret.py"", line 31, in res = govRet() File ""/home/ubuntu/dev/py36/python/ev/model/api3/returns.py"", line 56, in __ca ll__ decompose=self.decompose)) File ""/home/ubuntu/dev/py36/python/ev/model/returns/returnsGenerator.py"", line 70, in calc_returns dfs_data = self.mongo_dfs.get_data(mats=[1,mat,mat-1]) File ""/home/ubuntu/dev/py36/python/ev/model/api3/dfs.py"", line 262, in get_dat a record = self.mdb.load_unique_document(self.dfs_collection_name, spec) File ""/home/ubuntu/dev/py36/python/ev/model/api3/sim_io.py"", line 1109, in load_unique_document return self.collections[collection].load_unique_document(query, *args, **kwargs) File ""/home/ubuntu/dev/py36/python/ev/model/api3/sim_io.py"", line 501, in load_unique_document doc['data'] = ar1.load().values File ""/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/xarray/core/dataarray.py"", line 631, in load ds = self._to_temp_dataset().load(**kwargs) File ""/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/xarray/core/dataset.py"", line 494, in load evaluated_data = da.compute(*lazy_data.values(), **kwargs) File ""/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/dask/base.py"", line 398, in compute results = schedule(dsk, keys, **kwargs) File ""/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/dask/threaded.py"", line 76, in get pack_exception=pack_exception, **kwargs) pack_exception=pack_exception, **kwargs) File ""/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/dask/local .py"", line 459, in get_async raise_exception(exc, tb) File ""/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/dask/compa tibility.py"", line 112, in reraise raise exc File ""/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/dask/local .py"", line 230, in execute_task result = _execute_task(task, data) File ""/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/dask/core. py"", line 119, in _execute_task return func(*args2) File ""/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/dask/array /core.py"", line 82, in getter c = np.asarray(c) File ""/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/numpy/core /numeric.py"", line 501, in asarray return array(a, dtype, copy=False, order=order) File ""/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/xarray/cor e/indexing.py"", line 602, in __array__ return np.asarray(self.array, dtype=dtype) File ""/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/numpy/core/numeric.py"", line 501, in asarray return array(a, dtype, copy=False, order=order) File ""/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/xarray/core/indexing.py"", line 508, in __array__ return np.asarray(array[self.key], dtype=None) File ""/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/xarray/backends/netCDF4_.py"", line 64, in __getitem__ self._getitem) File ""/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/xarray/core/indexing.py"", line 776, in explicit_indexing_adapter result = raw_indexing_method(raw_key.tuple) File ""/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/xarray/backends/netCDF4_.py"", line 76, in _getitem array = getitem(original_array, key) File ""netCDF4/_netCDF4.pyx"", line 4095, in netCDF4._netCDF4.Variable.__getitem__ File ""netCDF4/_netCDF4.pyx"", line 3798, in netCDF4._netCDF4.Variable.shape.__get__ File ""netCDF4/_netCDF4.pyx"", line 3746, in netCDF4._netCDF4.Variable._getdims File ""netCDF4/_netCDF4.pyx"", line 1754, in netCDF4._netCDF4._ensure_nc_success RuntimeError: NetCDF: Not a valid ID ```
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/5276/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue