issue_comments

We didn't do a good job answering this (thank you @TomNicholas for responding), but likely the moment has passed, so I'll close. Please reopen with a MCVE if there's still an issue.

Hi there - are you able to narrow down the problem a little bit for us? Can you get the same error by opening any one file individually with xarray.open_dataset()? What if you use the netCDF4 library to access a variable directly without xarray? Failing that can you give us a file/files for which this fails, so that we can reproduce the problem locally? Also, have you tried using a more recent version of netCDF4? The most recent version on PyPI seems to be 1.5.6, but you're using 1.4.2.

As mentioned above, the error only happens after successful call the func for a few times, i.e, first few calls it works fine as expected. "Can you get the same error by opening any one file individually with `xarray.open_dataset()" Yes, I can. But the whole point is to use open_mfdataset.

I was able to generate a small code to re-produce the error. ```python import numpy as np import xarray as xr from copy import deepcopy import random import os import logging log = logging.getLogger(name)

def generate_data(xpath, m=10000,n=900,k=601): """

:param m:
:param n:
:param k:
:return:
"""
coord1=dict(dim1=range(m), dim2=range(n))
for mat in range(k):
    a = np.random.rand(m,n)
    file_name = "{:04d}.nc".format(mat)
    full_file_name = os.path.join(xpath, file_name)
    coord1['maturity'] = [mat]
    a = np.reshape(a,[m,n,1])
    ar = xr.DataArray(a, dims=['dim1', 'dim2', 'maturity'], coords=coord1)
    ar.to_netcdf(full_file_name)

def func(xpath, spec): log.debug("xpath=%s, spec=%s", xpath, spec)

doc = deepcopy(spec)

with xr.open_mfdataset(xpath + "/*.nc", concat_dim='maturity') as data:
        var_name=  list(data.data_vars)[0]
        ar = data[var_name]
        maturity = spec['maturity']
        ann = ar.cumsum(dim='maturity')
        ann = ann - 1
        ar1 = ann.sel(maturity=maturity)
        doc['data'] = ar1.load().values
return doc

def repeat_access_data_generate_error(xpath, k=601, l=1000): """ Dummy to generate error :return: """ spec={} dum=0.0 for _l1 in range(l): try: spec['maturity'] = np.random.randint(0,k,1)[0] doc = func(xpath, spec) #do dummy operation with doc dum = (dum + np.sum(doc['data']))/2.0 except Exception as err: log.error("spec=%s, err=%s", spec, err, exc_info=True)

return dum

if name == "main": FORMAT = '%(asctime)-15s, %(levelname)s %(process)d, %(filename)s:%(lineno)s - %(funcName)20s(), %(message)s' log_file_name = "errors.log" logging.basicConfig(filename=log_file_name, filemode='a+', level=logging.DEBUG, format=FORMAT) xpath="/home/ubuntu/runs/xrdata/" # folder to hold files # change values of m,n,k to re-produce the error generate_data(xpath,m=1000) repeat_access_data_generate_error(xpath)

```

Hi there - are you able to narrow down the problem a little bit for us? Can you get the same error by opening any one file individually with xarray.open_dataset()? What if you use the netCDF4 library to access a variable directly without xarray? Failing that can you give us a file/files for which this fails, so that we can reproduce the problem locally? Also, have you tried using a more recent version of netCDF4? The most recent version on PyPI seems to be 1.5.6, but you're using 1.4.2.