html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue https://github.com/pydata/xarray/issues/5706#issuecomment-1012204673,https://api.github.com/repos/pydata/xarray/issues/5706,1012204673,IC_kwDOAMm_X848VQSB,8291800,2022-01-13T14:48:08Z,2022-01-13T14:48:08Z,CONTRIBUTOR,"Sounds good, but it seems like you're correct that it's a netcdf/netcdf4-python problem here, so I'll defer to others as to what the best changes to default settings would be to avoid the segfaults","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,970619131 https://github.com/pydata/xarray/issues/5706#issuecomment-1012132794,https://api.github.com/repos/pydata/xarray/issues/5706,1012132794,IC_kwDOAMm_X848U-u6,8291800,2022-01-13T13:23:45Z,2022-01-13T13:23:45Z,CONTRIBUTOR,"ah sorry, didn't see the request for `ncdump`. ``` $ ncdump test_str_list.h5 netcdf test_str_list { dimensions: phony_dim_0 = 2 ; phony_dim_1 = 2 ; variables: string pairs(phony_dim_0, phony_dim_1) ; data: pairs = ""2020010120200201 "", NIL, ""2020010120200301 "", NIL ; } ``` Interesting that my `pairs` seems different than yours without the obvious trailing garbage. Also, when I run your first code snippet, I have different areas that are garbled, with both `NULLPAD` and `numpy_S` displaying garbage ``` netcdf test_str_list_attr { // global attributes: string :NULLPAD = ""20200101�, ('/home/scott/testdata.nc',), 'a', (('invalid_netcdf', None),)] During handling of the above exception, another exception occurred: Traceback (most recent call last): File ""testxr.py"", line 35, in da_new.to_dataset(name=""new_testdata"").to_netcdf(""testdata.nc"", engine=""h5netcdf"") File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/core/dataset.py"", line 1900, in to_netcdf return to_netcdf( File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/backends/api.py"", line 1060, in to_netcdf store = store_open(target, mode, format, group, **kwargs) File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/backends/h5netcdf_.py"", line 178, in open return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose) File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/backends/h5netcdf_.py"", line 123, in __init__ self._filename = find_root_and_group(self.ds)[0].filename File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/backends/h5netcdf_.py"", line 189, in ds return self._acquire() File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/backends/h5netcdf_.py"", line 181, in _acquire with self._manager.acquire_context(needs_lock) as root: File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/contextlib.py"", line 113, in __enter__ return next(self.gen) File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/backends/file_manager.py"", line 187, in acquire_context file, cached = self._acquire_with_cache_info(needs_lock) File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/backends/file_manager.py"", line 205, in _acquire_with_cache_info file = self._opener(*self._args, **kwargs) File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/h5netcdf/core.py"", line 712, in __init__ self._h5file = h5py.File(path, mode, **kwargs) File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/h5py/_hl/files.py"", line 406, in __init__ fid = make_fid(name, mode, userblock_size, File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/h5py/_hl/files.py"", line 179, in make_fid fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl) File ""h5py/_objects.pyx"", line 54, in h5py._objects.with_phil.wrapper File ""h5py/_objects.pyx"", line 55, in h5py._objects.with_phil.wrapper File ""h5py/h5f.pyx"", line 108, in h5py.h5f.create OSError: Unable to create file (unable to lock file, errno = 11, error message = 'Resource temporarily unavailable') ``` And here's the original ```$ python testxr.py Traceback (most recent call last): File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/backends/file_manager.py"", line 199, in _acquire_with_cache_info file = self._cache[self._key] File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/backends/lru_cache.py"", line 53, in __getitem__ value = self._cache[key] KeyError: [, ('/home/scott/testdata.nc',), 'a', (('invalid_netcdf', None),)] During handling of the above exception, another exception occurred: Traceback (most recent call last): File ""testxr.py"", line 35, in da_new.to_dataset(name=""new_testdata"").to_netcdf(""testdata.nc"", engine=""h5netcdf"") File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/core/dataset.py"", line 1900, in to_netcdf return to_netcdf( File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/backends/api.py"", line 1060, in to_netcdf store = store_open(target, mode, format, group, **kwargs) File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/backends/h5netcdf_.py"", line 178, in open return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose) File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/backends/h5netcdf_.py"", line 123, in __init__ self._filename = find_root_and_group(self.ds)[0].filename File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/backends/h5netcdf_.py"", line 189, in ds return self._acquire() File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/backends/h5netcdf_.py"", line 181, in _acquire with self._manager.acquire_context(needs_lock) as root: File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/contextlib.py"", line 113, in __enter__ return next(self.gen) File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/backends/file_manager.py"", line 187, in acquire_context file, cached = self._acquire_with_cache_info(needs_lock) File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/backends/file_manager.py"", line 205, in _acquire_with_cache_info file = self._opener(*self._args, **kwargs) File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/h5netcdf/core.py"", line 712, in __init__ self._h5file = h5py.File(path, mode, **kwargs) File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/h5py/_hl/files.py"", line 406, in __init__ fid = make_fid(name, mode, userblock_size, File ""/home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/h5py/_hl/files.py"", line 179, in make_fid fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl) File ""h5py/_objects.pyx"", line 54, in h5py._objects.with_phil.wrapper File ""h5py/_objects.pyx"", line 55, in h5py._objects.with_phil.wrapper File ""h5py/h5f.pyx"", line 108, in h5py.h5f.create OSError: Unable to create file (unable to lock file, errno = 11, error message = 'Resource temporarily unavailable') ``` `xr.show_versions()`
>>> xarray.show_versions() INSTALLED VERSIONS ------------------ commit: None python: 3.8.5 (default, Sep 4 2020, 07:30:14) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 3.10.0-1127.19.1.el7.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.10.6 libnetcdf: 4.8.0 xarray: 0.19.0 pandas: 1.1.2 numpy: 1.21.2 scipy: 1.6.1 netCDF4: 1.5.7 pydap: None h5netcdf: 0.11.0 h5py: 2.10.0 Nio: None zarr: 2.8.3 cftime: 1.2.1 nc_time_axis: None PseudoNetCDF: None rasterio: 1.2.8 cfgrib: 0.9.8.5 iris: None bottleneck: 1.3.2 dask: 2.12.0 distributed: 2.25.0 matplotlib: 3.3.1 cartopy: 0.20.0 seaborn: None numbagg: None pint: 0.17 setuptools: 49.6.0.post20200814 pip: 20.0.2 conda: 4.8.4 pytest: None IPython: 7.18.1 sphinx: 4.0.2
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1056881922 https://github.com/pydata/xarray/issues/5706#issuecomment-1011556328,https://api.github.com/repos/pydata/xarray/issues/5706,1011556328,IC_kwDOAMm_X848Sx_o,8291800,2022-01-12T23:51:07Z,2022-01-12T23:53:01Z,CONTRIBUTOR,"sure! here it is: ```bash $ h5dump test_str_list.h5 HDF5 ""test_str_list.h5"" { GROUP ""/"" { DATASET ""pairs"" { DATATYPE H5T_STRING { STRSIZE 8; STRPAD H5T_STR_NULLPAD; CSET H5T_CSET_ASCII; CTYPE H5T_C_S1; } DATASPACE SIMPLE { ( 2, 2 ) / ( 2, 2 ) } DATA { (0,0): ""20200101"", ""20200201"", (1,0): ""20200101"", ""20200301"" } } } } ``` (and just to include the specific traceback that hapened now, in case my versions are different from what I showed):
In [4]: import h5py ...: import xarray as xr ...: ...: with h5py.File(""test_str_list.h5"", ""w"") as hf: ...: hf[""pairs""] = np.array([[""20200101"", ""20200201""], [""20200101"", ""20200301""]]).astype(""S"") ...: ...: ds = xr.load_dataset(""test_str_list.h5"") /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/xarray/backends/plugins.py:68: RuntimeWarning: Engine 'cfgrib' loading failed: /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/gribapi/_bindings.cpython-38-x86_64-linux-gnu.so: undefined symbol: codes_bufr_key_is_header warnings.warn(f""Engine {name!r} loading failed:\n{ex}"", RuntimeWarning) /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/fsspec/implementations/local.py:29: FutureWarning: The default value of auto_mkdir=True has been deprecated and will be changed to auto_mkdir=False by default in a future release. warnings.warn( *** Error in `/home/scott/miniconda3/envs/mapping/bin/python': free(): invalid next size (fast): 0x00005564b64622a0 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x81679)[0x7f56e752b679] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/netCDF4/../../../libnetcdf.so.18(nc_free_string+0x25)[0x7f54cf53d1a5] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/netCDF4/_netCDF4.cpython-38-x86_64-linux-gnu.so(+0xcf3c8)[0x7f54cf7313c8] /home/scott/miniconda3/envs/mapping/bin/python(PyCFunction_Call+0x54)[0x5564b397df44] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/netCDF4/_netCDF4.cpython-38-x86_64-linux-gnu.so(+0x224fd)[0x7f54cf6844fd] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/netCDF4/_netCDF4.cpython-38-x86_64-linux-gnu.so(+0x559d9)[0x7f54cf6b79d9] /home/scott/miniconda3/envs/mapping/bin/python(PyObject_GetItem+0x45)[0x5564b39d7935] /home/scott/miniconda3/envs/mapping/bin/python(+0x128e0b)[0x5564b397ae0b] /home/scott/miniconda3/envs/mapping/bin/python(_PyEval_EvalFrameDefault+0x947)[0x5564b3a1ec77] /home/scott/miniconda3/envs/mapping/bin/python(+0x1b0736)[0x5564b3a02736] /home/scott/miniconda3/envs/mapping/bin/python(_PyEval_EvalFrameDefault+0x947)[0x5564b3a1ec77] /home/scott/miniconda3/envs/mapping/bin/python(_PyFunction_Vectorcall+0x1a6)[0x5564b3a01fc6] /home/scott/miniconda3/envs/mapping/bin/python(_PyEval_EvalFrameDefault+0x4e03)[0x5564b3a23133] /home/scott/miniconda3/envs/mapping/bin/python(_PyFunction_Vectorcall+0x1a6)[0x5564b3a01fc6] /home/scott/miniconda3/envs/mapping/bin/python(+0x1800cd)[0x5564b39d20cd] /home/scott/miniconda3/envs/mapping/bin/python(PyObject_GetItem+0x45)[0x5564b39d7935] /home/scott/miniconda3/envs/mapping/bin/python(_PyEval_EvalFrameDefault+0xd53)[0x5564b3a1f083] /home/scott/miniconda3/envs/mapping/bin/python(_PyEval_EvalCodeWithName+0x2c3)[0x5564b3a00db3] /home/scott/miniconda3/envs/mapping/bin/python(_PyFunction_Vectorcall+0x378)[0x5564b3a02198] /home/scott/miniconda3/envs/mapping/bin/python(+0x1b0841)[0x5564b3a02841] /home/scott/miniconda3/envs/mapping/bin/python(+0x12404d)[0x5564b397604d] /home/scott/miniconda3/envs/mapping/bin/python(_PyObject_CallFunction_SizeT+0x99)[0x5564b39761f9] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so(+0xa11fd)[0x7f56dddfe1fd] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so(+0xa54d7)[0x7f56dde024d7] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so(+0x8a2d5)[0x7f56ddde72d5] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so(+0x8adc4)[0x7f56ddde7dc4] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so(+0xa559a)[0x7f56dde0259a] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so(+0xa5ac9)[0x7f56dde02ac9] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so(+0x13f2b7)[0x7f56dde9c2b7] /home/scott/miniconda3/envs/mapping/bin/python(+0x129082)[0x5564b397b082] /home/scott/miniconda3/envs/mapping/bin/python(_PyEval_EvalFrameDefault+0x181e)[0x5564b3a1fb4e] /home/scott/miniconda3/envs/mapping/bin/python(_PyEval_EvalCodeWithName+0x2c3)[0x5564b3a00db3] /home/scott/miniconda3/envs/mapping/bin/python(_PyFunction_Vectorcall+0x378)[0x5564b3a02198] /home/scott/miniconda3/envs/mapping/bin/python(+0x1b0841)[0x5564b3a02841] /home/scott/miniconda3/envs/mapping/bin/python(+0x12404d)[0x5564b397604d] /home/scott/miniconda3/envs/mapping/bin/python(_PyObject_CallFunction_SizeT+0x99)[0x5564b39761f9] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so(+0xa11fd)[0x7f56dddfe1fd] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so(+0xa54d7)[0x7f56dde024d7] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so(+0x8a2d5)[0x7f56ddde72d5] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so(+0x8adc4)[0x7f56ddde7dc4] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so(+0xa559a)[0x7f56dde0259a] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so(+0xa5ac9)[0x7f56dde02ac9] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so(+0x13f2b7)[0x7f56dde9c2b7] /home/scott/miniconda3/envs/mapping/bin/python(+0x129082)[0x5564b397b082] /home/scott/miniconda3/envs/mapping/bin/python(_PyEval_EvalFrameDefault+0x4e03)[0x5564b3a23133] /home/scott/miniconda3/envs/mapping/bin/python(_PyFunction_Vectorcall+0x1a6)[0x5564b3a01fc6] /home/scott/miniconda3/envs/mapping/bin/python(_PyEval_EvalFrameDefault+0xa63)[0x5564b3a1ed93] /home/scott/miniconda3/envs/mapping/bin/python(_PyEval_EvalCodeWithName+0x2c3)[0x5564b3a00db3] /home/scott/miniconda3/envs/mapping/bin/python(_PyFunction_Vectorcall+0x378)[0x5564b3a02198] /home/scott/miniconda3/envs/mapping/bin/python(+0x1b0841)[0x5564b3a02841] /home/scott/miniconda3/envs/mapping/bin/python(+0x12404d)[0x5564b397604d] /home/scott/miniconda3/envs/mapping/bin/python(_PyObject_CallFunction_SizeT+0x99)[0x5564b39761f9] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so(+0xa11fd)[0x7f56dddfe1fd] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so(+0xa54d7)[0x7f56dde024d7] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so(+0x8a2d5)[0x7f56ddde72d5] /home/scott/miniconda3/envs/mapping/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so Aborted (core dumped)
xr.show_versions
In [2]: xr.show_versions() INSTALLED VERSIONS ------------------ commit: None python: 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:59:51) [GCC 9.4.0] python-bits: 64 OS: Linux OS-release: 3.10.0-1062.4.1.el7.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.10.6 libnetcdf: 4.7.4 xarray: 0.20.2 pandas: 1.1.0 numpy: 1.21.2 scipy: 1.5.3 netCDF4: 1.5.4 pydap: None h5netcdf: 0.11.0 h5py: 3.2.1 Nio: None zarr: 2.8.3 cftime: 1.2.1 nc_time_axis: None PseudoNetCDF: None rasterio: 1.2.6 cfgrib: None iris: None bottleneck: 1.3.2 dask: 2021.01.0 distributed: 2.20.0 matplotlib: 3.3.1 cartopy: 0.19.0.post1 seaborn: None numbagg: None fsspec: 0.6.3 cupy: 9.0.0 pint: 0.17 sparse: None setuptools: 50.3.2 pip: 21.2.4 conda: 4.8.4 pytest: 6.2.4 IPython: 7.18.1 sphinx: 4.0.2
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,970619131 https://github.com/pydata/xarray/issues/5644#issuecomment-888522258,https://api.github.com/repos/pydata/xarray/issues/5644,888522258,IC_kwDOAMm_X8409cYS,8291800,2021-07-28T18:20:49Z,2021-07-28T18:20:49Z,CONTRIBUTOR,"As a temporary workout for my case, I'm just going to do ```python In [3]: pf = (da.copy(True)).polyfit(""z"", deg=2, w=np.arange(nz)) In [5]: da.max(), da.mean() Out[5]: ( array(0.99878237), array(0.50869358)) ``` The thing I don't understand is that `_to_temp_dataset` seems to be trying to do a deep copy (based on the argument names) https://github.com/pydata/xarray/blob/da99a5664df4f5013c2f6b0e758394bec5e0bc80/xarray/core/dataarray.py#L490-L491 But it is acting like a shallow copy: ```python In [6]: pf = (da.copy(False)).polyfit(""z"", deg=2, w=np.arange(nz)) In [7]: da.max(), da.mean() Out[7]: ( array(8.92217147), array(2.29014397)) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,955043280