id,node_id,number,title,user,state,locked,assignee,milestone,comments,created_at,updated_at,closed_at,author_association,active_lock_reason,draft,pull_request,body,reactions,performed_via_github_app,state_reason,repo,type
1596115847,I_kwDOAMm_X85fIsuH,7549,HDF5-DIAG warnings calling `open_mfdataset` with more than `file_cache_maxsize` datasets (hdf5 1.12.2),132147,open,0,,,10,2023-02-23T02:28:23Z,2023-03-26T00:41:11Z,,CONTRIBUTOR,,,,"### What happened?
Using `ds = open_mfdataset(...)` to open more than `file_cache_maxsize` files, then saving the combined dataset with `ds.to_netcdf()` prints many HDF5-DIAG warnings. This happens when using `hdf5==1.12.2` as bundled with `netcdf4~=1.6.0`. Downgrading to `netcdf4~=1.5.8` which bundles `hdf5==1.12.0` stops the warnings being printed. The output file appears to be correct in either case.
### What did you expect to happen?
No warnings from HDF5-DIAG. Either raise an error because of the number of files being opened at once, or behaving as `open_mfdataset()` normally does.
### Minimal Complete Verifiable Example
```Python
import argparse
import pathlib
import tempfile
from typing import List
import netCDF4
import xarray
HERE = pathlib.Path(__file__).parent
def add_arguments(parser: argparse.ArgumentParser):
parser.add_argument('count', type=int, default=200, nargs='?')
parser.add_argument('--file-cache-maxsize', type=int, required=False)
def main():
parser = argparse.ArgumentParser()
add_arguments(parser)
opts = parser.parse_args()
if opts.file_cache_maxsize is not None:
xarray.set_options(file_cache_maxsize=opts.file_cache_maxsize)
temp_dir = tempfile.mkdtemp(dir=HERE, prefix='work-dir-')
work_dir = pathlib.Path(temp_dir)
print(""Working in"", work_dir.name)
print(""Making"", opts.count, ""datasets"")
dataset_paths = make_many_datasets(work_dir, count=opts.count)
print(""Combining"", len(dataset_paths), ""datasets"")
dataset = xarray.open_mfdataset(dataset_paths, lock=False)
dataset.to_netcdf(work_dir / 'combined.nc')
def make_many_datasets(
work_dir: pathlib.Path,
count: int = 200
) -> List[pathlib.Path]:
dataset_paths = []
for i in range(count):
variable = f'var_{i}'
path = work_dir / f'{variable}.nc'
dataset_paths.append(path)
make_dataset(path, variable)
return dataset_paths
def make_dataset(
path: pathlib.Path,
variable: str,
) -> None:
ds = netCDF4.Dataset(path, ""w"", format=""NETCDF4"")
ds.createDimension(""x"", 1)
var = ds.createVariable(variable, ""i8"", (""x"",))
var[:] = 1
ds.close()
if __name__ == '__main__':
main()
```
### MVCE confirmation
- [x] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- [x] Complete example — the example is self-contained, including all data and the text of any traceback.
- [x] Verifiable example — the example copy & pastes into an IPython prompt or [Binder notebook](https://mybinder.org/v2/gh/pydata/xarray/main?urlpath=lab/tree/doc/examples/blank_template.ipynb), returning the result.
- [x] New issue — a search of GitHub Issues suggests this is not a duplicate.
### Relevant log output
```Python
Working in work-dir-yi3jp5ly
Making 128 datasets
Combining 128 datasets
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1:
#000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
major: Attribute
minor: Can't open object
#001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
major: Attribute
minor: Can't open object
#004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitGroomNumberOfSignificantDigits'
major: Attribute
minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1:
#000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
major: Attribute
minor: Can't open object
#001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
major: Attribute
minor: Can't open object
#004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'
major: Attribute
minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1:
#000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
major: Attribute
minor: Can't open object
#001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
major: Attribute
minor: Can't open object
#004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitRoundNumberOfSignificantBits'
major: Attribute
minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1:
#000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
major: Attribute
minor: Can't open object
#001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
major: Attribute
minor: Can't open object
#004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitGroomNumberOfSignificantDigits'
major: Attribute
minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1:
#000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
major: Attribute
minor: Can't open object
#001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
major: Attribute
minor: Can't open object
#004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'
major: Attribute
minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1:
#000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
major: Attribute
minor: Can't open object
#001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
major: Attribute
minor: Can't open object
#004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitRoundNumberOfSignificantBits'
major: Attribute
minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 2:
#000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
major: Attribute
minor: Can't open object
#001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
major: Attribute
minor: Can't open object
#004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitGroomNumberOfSignificantDigits'
major: Attribute
minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 2:
#000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
major: Attribute
minor: Can't open object
#001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
major: Attribute
minor: Can't open object
#004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'
major: Attribute
minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 2:
#000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
major: Attribute
minor: Can't open object
#001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
major: Attribute
minor: Can't open object
#004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitRoundNumberOfSignificantBits'
major: Attribute
minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 3:
#000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
major: Attribute
minor: Can't open object
#001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
major: Attribute
minor: Can't open object
#004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitGroomNumberOfSignificantDigits'
major: Attribute
minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 3:
#000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
major: Attribute
minor: Can't open object
#001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
major: Attribute
minor: Can't open object
#004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'
major: Attribute
minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 3:
#000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
major: Attribute
minor: Can't open object
#001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
major: Virtual Object Layer
minor: Can't open object
#003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
major: Attribute
minor: Can't open object
#004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
major: Attribute
minor: Unable to initialize object
#005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitRoundNumberOfSignificantBits'
major: Attribute
minor: Object not found
```
### Anything else we need to know?
The example is a script to run on the command line. Assuming the file is named `test.py`, invoke as:
```shell
# This will show the error. Defaults to making 200 files
$ python3 ./test.py
# This will not show the error - the number of files is less than file_cache_maxsize:
$ python3 ./test.py 127
# This will adjust file_cache_maxsize to show the error again, despite the lower number of files
$ python3 ./test.py 11 --file-cache-maxsize=10
```
The log output is from `python3 ./test.py 128`. The length of the log output depends on the number of files opened - the more files the longer the log. The log output from `python3 ./test.py` using the default of 200 datasets was too long to include in this issue!
All output files are restricted to directories created in the current working directory named `work-dir-XXXXXXX`, and are retained after the program exits. 200 datasets take up a total of 1.7MB.
### Environment
Failing environment:
INSTALLED VERSIONS
------------------
commit: None
python: 3.8.10 (default, Nov 14 2022, 12:59:47)
[GCC 9.4.0]
python-bits: 64
OS: Linux
OS-release: 5.15.0-58-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_AU.UTF-8
LOCALE: ('en_AU', 'UTF-8')
libhdf5: 1.12.2
libnetcdf: 4.9.0
xarray: 2023.1.0
pandas: 1.5.3
numpy: 1.24.2
scipy: None
netCDF4: 1.6.2
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.6.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2023.2.0
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2023.1.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 44.0.0
pip: 23.0.1
conda: None
pytest: None
mypy: None
IPython: None
sphinx: None
Working environment:
INSTALLED VERSIONS
------------------
commit: None
python: 3.8.10 (default, Nov 14 2022, 12:59:47)
[GCC 9.4.0]
python-bits: 64
OS: Linux
OS-release: 5.15.0-58-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_AU.UTF-8
LOCALE: ('en_AU', 'UTF-8')
libhdf5: 1.12.0
libnetcdf: 4.7.4
xarray: 2023.1.0
pandas: 1.5.3
numpy: 1.24.2
scipy: None
netCDF4: 1.5.8
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.6.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2023.2.0
distributed: None
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
fsspec: 2023.1.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 44.0.0
pip: 23.0.1
conda: None
pytest: None
mypy: None
IPython: None
sphinx: None
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7549/reactions"", ""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,issue