issues: 672262818

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
672262818 MDU6SXNzdWU2NzIyNjI4MTg= 4304 netCDF4 Resource unavailable error 45180714 closed 0     2 2020-08-03T18:28:28Z 2022-04-17T18:03:31Z 2022-04-17T18:03:31Z NONE      

I've been encountering the following error occasionally when running some rather heavy scripts. Sometimes the code will run fine (with the same data) and other times it will exit with this traceback: Error 11: Resource temporarily unavailable Traceback (most recent call last): File "globalmodels.py", line 213, in <module> run() File "globalmodels.py", line 210, in run M.make_maps(time) File "globalmodels.py", line 50, in make_maps d = TIGGEParser(ds) File "/EWALL_TEST/reforecast/utils/parsers.py", line 248, in __init__ field = ds.sel(level=lev)[var['name']].values * units(var['units']) File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/xarray/core/dataarray.py", line 558, in values return self.variable.values File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/xarray/core/variable.py", line 446, in values return _as_array_or_item(self._data) File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/xarray/core/variable.py", line 249, in _as_array_or_item data = np.asarray(data) File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/numpy/core/_asarray.py", line 85, in asarray return array(a, dtype, copy=False, order=order) File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/xarray/core/indexing.py", line 677, in __array__ self._ensure_cached() File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/xarray/core/indexing.py", line 674, in _ensure_cached self.array = NumpyIndexingAdapter(np.asarray(self.array)) File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/numpy/core/_asarray.py", line 85, in asarray return array(a, dtype, copy=False, order=order) File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/xarray/core/indexing.py", line 653, in __array__ return np.asarray(self.array, dtype=dtype) File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/numpy/core/_asarray.py", line 85, in asarray return array(a, dtype, copy=False, order=order) File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/xarray/core/indexing.py", line 557, in __array__ return np.asarray(array[self.key], dtype=None) File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/numpy/core/_asarray.py", line 85, in asarray return array(a, dtype, copy=False, order=order) File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/xarray/coding/variables.py", line 72, in __array__ return self.func(self.array) File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/xarray/coding/variables.py", line 218, in _scale_offset_decoding data = np.array(data, dtype=dtype, copy=True) File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/xarray/coding/variables.py", line 72, in __array__ return self.func(self.array) File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/xarray/coding/variables.py", line 138, in _apply_mask data = np.asarray(data, dtype=dtype) File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/numpy/core/_asarray.py", line 85, in asarray return array(a, dtype, copy=False, order=order) File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/xarray/core/indexing.py", line 557, in __array__ return np.asarray(array[self.key], dtype=None) File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 72, in __getitem__ return indexing.explicit_indexing_adapter( File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/xarray/core/indexing.py", line 837, in explicit_indexing_adapter result = raw_indexing_method(raw_key.tuple) File "/home/meteo/kps5442/.conda/envs/ewall/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 85, in _getitem array = getitem(original_array, key) File "netCDF4/_netCDF4.pyx", line 4408, in netCDF4._netCDF4.Variable.__getitem__ File "netCDF4/_netCDF4.pyx", line 5352, in netCDF4._netCDF4.Variable._get File "netCDF4/_netCDF4.pyx", line 1887, in netCDF4._netCDF4._ensure_nc_success RuntimeError: Resource temporarily unavailable The error occurs when I try to parse some information out of the dataset - I'm selecting based on level and then variable, and pulling out the values. From the error traceback, it seems like the values are causing the problem.

I'm not sure if it matters, but this seems to only be a problem when I'm using multiprocessing, when there could be a number of processes parsing different datasets at the same time. Sometimes, though, this isn't a problem at all, and the code runs as expected with no errors.

I'm not sure if this error is because of bad data, or limited resources. That would inform how I try to catch errors and re-try. I'm hoping someone here can point me in the right direction. 

{
    "url": "https://api.github.com/repos/pydata/xarray/issues/4304/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed 13221727 issue

Links from other tables