issue_comments

Since we have eliminated xarray with this, you should be able to submit an issue to the h5py issue tracker while mentioning that this is probably a bug in libhdf5 since netcdf4 also fails with the same error (and you can also link this issue for more information)

I had to change ints and floats to doubles to reproduce the issue. ```python import h5py

N_TIMES = 48 with h5py.File("/my_s3_fs/test.nc", mode="w") as f: time = f.create_dataset("time", (N_TIMES,), dtype="d") time[:] = 0

d1 = f.create_dataset("d1", (N_TIMES, 201, 201), dtype="d")
d1[:] = 0

```

great, good to know. Can you try this with h5py: ```python import h5py

N_TIMES = 48 with h5py.File("test.nc", mode="w") as f: time = f.create_dataset("time", (N_TIMES,), dtype="i") time[:] = 0

d1 = f.create_dataset("d1", (N_TIMES, 201, 201), dtype="f")
d1[:] = 0

```

datatype seems to be not important. But the two variables are required to get a segfault. The following with just floats produces a segfault ``` import numpy as np import xarray as xr

N_TIMES=48 ds = xr.Dataset({"time": ("T", np.zeros((N_TIMES))), 'd1': (["T", "x", "y"], np.zeros((N_TIMES, 201,201)))}) ds.to_netcdf(path="/my_s3_fs/test_netcdf.nc", format="NETCDF4", mode="w") ```

It seems that we need the time variable to reproduce the problem. The following code does not fail: ``` import numpy as np import xarray as xr import pandas as pd

N_TIMES=64 ds = xr.Dataset({'d1': (["T", "x", "y"], np.zeros((N_TIMES, 201,201)))}) ds.to_netcdf(path="/my_s3_fs/test_netcdf.nc", format="NETCDF4", mode="w") ```

okay, then does changing the dtype do anything? I.e. does this only happen with datetime64 / bytes, or do int / float / str also fail?

with this: python ds2 = ds.time.dt.strftime("%Y%m%d%H%M%S").str.encode("utf-8").to_dataset().assign(d1=ds.d1) but we don't really need to check if the first dataset already fails.

Now I'd probably check if it's just the size that makes it fail (i.e. remove "time" from ds and keep just d1 while maybe increasing it by one if it does not fail as-is), or if it depends on the dtype (i.e. replace set time_vals to np.arange(N_TIMES, dtype=int)).

The first one results in a segfault: ```python import numpy as np import xarray as xr import pandas as pd

N_TIMES = 48 time_vals = pd.date_range("2022-10-06", freq="20 min", periods=N_TIMES) ds = xr.Dataset({"time": ("T", time_vals), 'd1': (["T", "x", "y"], np.zeros((len(time_vals), 201,201)))}) ds.to_netcdf(path="/my_s3_fs/test_netcdf.nc", format="NETCDF4", mode="w") ``` Not sure how to add the 3D var to the second dataset.

which ones fail if you add the 3D variable?

Will try to reproduce this with h5py. For the bug to show up the file has to be large enough. That is why my example has a 2D array variable alongside the time dimension. With just the time dimension the script completes without an error. All three cases work without an error: ds.to_netcdf(), ds2.to_netcdf(), and ds3.to_netcdf()

if this crashes with both netcdf4 and h5netcdf this might be a bug in the libhdf5 library. If we can manage to reduce this to use just h5py (or netCDF4), it should be suitable for reporting on their issue tracker, and those libraries can then push it further to libhdf5 (otherwise, if you're up for investigating / debugging the C library, you could also report to libhdf5 directly).

As for the MCVE: I wonder if we can trim it a bit. Can you reproduce with ```python import xarray as xr import pandas as pd

N_TIMES = 48 time_vals = pd.date_range("2022-10-06", freq="20 min", periods=N_TIMES) ds = xr.Dataset({"time": ("T", time_vals)}) ds.to_netcdf(path="/my_s3_fs/test_netcdf.nc", format="NETCDF4", mode="w") or, if it is important to have bytes:python ds2 = ds.time.dt.strftime("%Y%m%d%H%M%S").str.encode("utf-8").to_dataset() ds2.to_netcdf(...) also it would be interesting to know if this happens only for data variables, or if coordinates have the same effect (use `ds2` instead of `ds2` if bytes are important):python ds3 = ds.set_coords("time") ds3.to_netcdf(...) ```

Adding a gdb stackrace.txt from corefile obtained with docker run -v /mnt/fs:/my_s3_fs -it --rm --ulimit core=-1 --privileged netcdf:latest /bin/bash and sudo sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t python mcve.py

Thanks @d1mach . Could it be related to https://github.com/pydata/xarray/issues/7136 ?

libnetcdf, netcdf4 and hdf5 are at their latest versions available on conda-forge

Can confirm the issue with xarray 2022.6.0 and dask 2022.9.2. The latest versions available on conda-forge. The issue might be related to netcdf4 and hdf5 libraries. Will try to update this as well.

That's quite an old version of xarray! Could we confirm it has similar results on a more recent version?