html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/5276#issuecomment-1094072451,https://api.github.com/repos/pydata/xarray/issues/5276,1094072451,IC_kwDOAMm_X85BNjiD,5635139,2022-04-09T15:49:50Z,2022-04-09T15:49:50Z,MEMBER,"We didn't do a good job answering this (thank you @TomNicholas for responding), but likely the moment has passed, so I'll close. Please reopen with a MCVE if there's still an issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,878481461
https://github.com/pydata/xarray/issues/5276#issuecomment-836869041,https://api.github.com/repos/pydata/xarray/issues/5276,836869041,MDEyOklzc3VlQ29tbWVudDgzNjg2OTA0MQ==,11815787,2021-05-10T15:50:37Z,2021-05-10T15:50:37Z,NONE,"> Hi there - are you able to narrow down the problem a little bit for us? Can you get the same error by opening any one file individually with `xarray.open_dataset()`? What if you use the `netCDF4` library to access a variable directly without xarray? Failing that can you give us a file/files for which this fails, so that we can reproduce the problem locally? Also, have you tried using a more recent version of netCDF4? The most recent version on PyPI seems to be 1.5.6, but you're using 1.4.2.
As mentioned above, the error only happens after successful call the func for a few times, i.e, first few calls it works fine as expected.
_""Can you get the same error by opening any one file individually with `xarray.open_dataset()""_
Yes, I can. But the whole point is to use open_mfdataset.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,878481461
https://github.com/pydata/xarray/issues/5276#issuecomment-836857104,https://api.github.com/repos/pydata/xarray/issues/5276,836857104,MDEyOklzc3VlQ29tbWVudDgzNjg1NzEwNA==,11815787,2021-05-10T15:41:56Z,2021-05-10T15:43:04Z,NONE,"I was able to generate a small code to re-produce the error.
```python
import numpy as np
import xarray as xr
from copy import deepcopy
import random
import os
import logging
log = logging.getLogger(__name__)
def generate_data(xpath, m=10000,n=900,k=601):
""""""
:param m:
:param n:
:param k:
:return:
""""""
coord1=dict(dim1=range(m), dim2=range(n))
for mat in range(k):
a = np.random.rand(m,n)
file_name = ""{:04d}.nc"".format(mat)
full_file_name = os.path.join(xpath, file_name)
coord1['maturity'] = [mat]
a = np.reshape(a,[m,n,1])
ar = xr.DataArray(a, dims=['dim1', 'dim2', 'maturity'], coords=coord1)
ar.to_netcdf(full_file_name)
def func(xpath, spec):
log.debug(""xpath=%s, spec=%s"", xpath, spec)
doc = deepcopy(spec)
with xr.open_mfdataset(xpath + ""/*.nc"", concat_dim='maturity') as data:
var_name= list(data.data_vars)[0]
ar = data[var_name]
maturity = spec['maturity']
ann = ar.cumsum(dim='maturity')
ann = ann - 1
ar1 = ann.sel(maturity=maturity)
doc['data'] = ar1.load().values
return doc
def repeat_access_data_generate_error(xpath, k=601, l=1000):
""""""
Dummy to generate error
:return:
""""""
spec={}
dum=0.0
for _l1 in range(l):
try:
spec['maturity'] = np.random.randint(0,k,1)[0]
doc = func(xpath, spec)
#do dummy operation with doc
dum = (dum + np.sum(doc['data']))/2.0
except Exception as err:
log.error(""spec=%s, err=%s"", spec, err, exc_info=True)
return dum
if __name__ == ""__main__"":
FORMAT = '%(asctime)-15s, %(levelname)s %(process)d, %(filename)s:%(lineno)s - %(funcName)20s(), %(message)s'
log_file_name = ""errors.log""
logging.basicConfig(filename=log_file_name, filemode='a+', level=logging.DEBUG, format=FORMAT)
xpath=""/home/ubuntu/runs/xrdata/"" # folder to hold files
# change values of m,n,k to re-produce the error
generate_data(xpath,m=1000)
repeat_access_data_generate_error(xpath)
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,878481461
https://github.com/pydata/xarray/issues/5276#issuecomment-834600113,https://api.github.com/repos/pydata/xarray/issues/5276,834600113,MDEyOklzc3VlQ29tbWVudDgzNDYwMDExMw==,35968931,2021-05-07T16:32:08Z,2021-05-07T16:32:08Z,MEMBER,"Hi there - are you able to narrow down the problem a little bit for us? Can you get the same error by opening any one file individually with `xarray.open_dataset()`? What if you use the `netCDF4` library to access a variable directly without xarray? Failing that can you give us a file/files for which this fails, so that we can reproduce the problem locally? Also, have you tried using a more recent version of netCDF4? The most recent version on PyPI seems to be 1.5.6, but you're using 1.4.2.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,878481461