issue_comments: 117993960

OK... as a follow-up, I did some tests and with netcdf4 I got this error again, but using open_mfdataset with the latest versions of h5py & h5netcdf I don't. But there are some decodings that aren't happening now... for whatever reason (maybe h5netcdf?). Anyway, my netcdf files store the attributes in 'ascii', that is, bytes in python so when trying to check for the time I get:

Traceback (most recent call last): File "segfault.py", line 62, in <module> concat_dim='time', engine='h5netcdf')) File "/ichec/home/users/razvan/.local/lib/python3.4/site-packages/xray/backends/api.py", line 202, in open_mfdataset datasets = [open_dataset(p, **kwargs) for p in paths] File "/ichec/home/users/razvan/.local/lib/python3.4/site-packages/xray/backends/api.py", line 202, in <listcomp> datasets = [open_dataset(p, **kwargs) for p in paths] File "/ichec/home/users/razvan/.local/lib/python3.4/site-packages/xray/backends/api.py", line 145, in open_dataset return maybe_decode_store(store) File "/ichec/home/users/razvan/.local/lib/python3.4/site-packages/xray/backends/api.py", line 101, in maybe_decode_store concat_characters=concat_characters, decode_coords=decode_coords) File "/ichec/home/users/razvan/.local/lib/python3.4/site-packages/xray/conventions.py", line 850, in decode_cf decode_coords) File "/ichec/home/users/razvan/.local/lib/python3.4/site-packages/xray/conventions.py", line 791, in decode_cf_variables decode_times=decode_times) File "/ichec/home/users/razvan/.local/lib/python3.4/site-packages/xray/conventions.py", line 735, in decode_cf_variable if 'since' in attributes['units']: TypeError: Type str doesn't support the buffer API

This is simple to solve.. just have every byte attribute decode to 'utf8' when first reading in the variables... I'll have some more time to look at this alter today.

edit: boy... there are some differences between these packages (netcdf4 & h5netcdf)... so, when trying to open_mfdataset with netcdf4 I get the segmentation fault... when I open it with h5netcdf I don't, but I the attributes are in bytes so then xray gives some errors when trying to get the date/time... but netcdf4 doesn't produce this error, it probably converts the bytes to strings internally... so I went in and tried to patch some .decode('utf8') here and there in xray and it works... when using h5netcdf, but then I get another error from h5netcdf:

File "/ichec/home/users/razvan/.local/lib/python3.4/site-packages/h5py/_hl/attrs.py", line 55, in __getitem__ raise IOError("Empty attributes cannot be read") OSError: Empty attributes cannot be read

I didn't put the full error cause I don't think it's relevant. Anyway, needless to say... netcdf4 doesn't give this error... so these things need to be put in accordance somehow :)

edit2: so I was going through the posts here and now I saw you addressed this issue using that lock thing, which is set to True by default in open_datset, right? well, I don't know exactly what this thing is supposed to do, but I'm still getting a segmentation fault, but as stated before, only when using netcdf4, not h5netcdf, but then I run in that inconsistency with the ascii vs utf8 issue if I use h5netcdf... maybe I should open an open issue about this string issue? I don't know if this is an upstream issue or not, I mean, I guess h5netcdf just decides to not convert the ascii to utf8, whereas netcdf4 goes with the more contemporary approach of returning utf8... or is this internally handled by xray?