issues: 878481461

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
878481461 MDU6SXNzdWU4Nzg0ODE0NjE= 5276 open_mfdataset: Not a valid ID 11815787 closed 0     4 2021-05-07T05:34:02Z 2022-04-09T15:49:50Z 2022-04-09T15:49:50Z NONE      

I have about 601 NETCDF4 files saved using xarray. We try to use open_mfdataset to access these files. The main code calls this function many times. At the first few calls, it works fine, after for a while it throw the following error message "RuntimeError: NetCDF: Not a valid ID"

python def func(xpath, spec): doc = deepcopy(spec) with xr.open_mfdataset(xpath + "/*.nc", concat_dim='maturity') as data: var_name= list(data.data_vars)[0] ar = data[var_name] maturity = spec['maturity'] ann = ar.cumsum(dim='maturity') ann = ann - 1 ar1 = ann.sel(maturity=maturity) doc['data'] = ar1.load().values return doc

Environment:

Output of <tt>xr.show_versions()</tt> INSTALLED VERSIONS ------------------ commit: None python: 3.6.8.final.0 python-bits: 64 OS: Linux OS-release: 5.4.0-1047-aws machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: C.UTF-8 LOCALE: en_US.UTF-8 xarray: 0.11.0 pandas: 0.24.1 numpy: 1.15.4 scipy: 1.2.0 netCDF4: 1.4.2 h5netcdf: None h5py: 2.9.0 Nio: None zarr: None cftime: 1.0.3.4 PseudonetCDF: None rasterio: None iris: None bottleneck: 1.2.1 cyordereddict: None dask: 1.1.1 distributed: 1.25.3 matplotlib: 3.0.2 cartopy: None seaborn: 0.9.0 setuptools: 40.7.3 pip: 19.0.1 conda: None pytest: 4.2.0 IPython: 7.1.1 sphinx: 1.8.4

This error also happens with xarray version 0.10.9

Error trace:

```python 2021-05-05 09:28:19,911, DEBUG 7621, sim_io.py:483 - load_unique_document(), xpa th=/home/ubuntu/runs/20210331_001/nominal_dfs/uk 2021-05-05 09:28:42,774, ERROR 7621, run_gov_ret.py:33 - <module>(), Unknown error=NetCDF: Not a valid ID Traceback (most recent call last): File "/home/ubuntu/dev/py36/python/ev/model/api3/run_gov_ret.py", line 31, in <module> res = govRet() File "/home/ubuntu/dev/py36/python/ev/model/api3/returns.py", line 56, in __ca ll__ decompose=self.decompose)) File "/home/ubuntu/dev/py36/python/ev/model/returns/returnsGenerator.py", line 70, in calc_returns dfs_data = self.mongo_dfs.get_data(mats=[1,mat,mat-1]) File "/home/ubuntu/dev/py36/python/ev/model/api3/dfs.py", line 262, in get_dat a record = self.mdb.load_unique_document(self.dfs_collection_name, spec) File "/home/ubuntu/dev/py36/python/ev/model/api3/sim_io.py", line 1109, in load_unique_document return self.collections[collection].load_unique_document(query, *args, **kwargs) File "/home/ubuntu/dev/py36/python/ev/model/api3/sim_io.py", line 501, in load_unique_document doc['data'] = ar1.load().values File "/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/xarray/core/dataarray.py", line 631, in load ds = self._to_temp_dataset().load(**kwargs) File "/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/xarray/core/dataset.py", line 494, in load evaluated_data = da.compute(*lazy_data.values(), **kwargs) File "/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/dask/base.py", line 398, in compute results = schedule(dsk, keys, **kwargs) File "/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/dask/threaded.py", line 76, in get pack_exception=pack_exception, **kwargs) pack_exception=pack_exception, **kwargs) File "/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/dask/local .py", line 459, in get_async raise_exception(exc, tb) File "/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/dask/compa tibility.py", line 112, in reraise raise exc File "/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/dask/local .py", line 230, in execute_task result = _execute_task(task, data) File "/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/dask/core. py", line 119, in _execute_task return func(*args2) File "/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/dask/array /core.py", line 82, in getter c = np.asarray(c) File "/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/numpy/core /numeric.py", line 501, in asarray return array(a, dtype, copy=False, order=order) File "/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/xarray/cor e/indexing.py", line 602, in __array__ return np.asarray(self.array, dtype=dtype) File "/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/numpy/core/numeric.py", line 501, in asarray return array(a, dtype, copy=False, order=order) File "/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/xarray/core/indexing.py", line 508, in __array__ return np.asarray(array[self.key], dtype=None) File "/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 64, in __getitem__ self._getitem) File "/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/xarray/core/indexing.py", line 776, in explicit_indexing_adapter result = raw_indexing_method(raw_key.tuple) File "/home/ubuntu/miniconda3/envs/egan/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 76, in _getitem array = getitem(original_array, key) File "netCDF4/_netCDF4.pyx", line 4095, in netCDF4._netCDF4.Variable.__getitem__ File "netCDF4/_netCDF4.pyx", line 3798, in netCDF4._netCDF4.Variable.shape.__get__ File "netCDF4/_netCDF4.pyx", line 3746, in netCDF4._netCDF4.Variable._getdims File "netCDF4/_netCDF4.pyx", line 1754, in netCDF4._netCDF4._ensure_nc_success RuntimeError: NetCDF: Not a valid ID ``` 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/5276/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed 13221727 issue

Links from other tables