issues: 597657663

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
597657663 MDU6SXNzdWU1OTc2NTc2NjM= 3961 Hangs while saving netcdf file opened using xr.open_mfdataset with lock=None 1159930 open 0     13 2020-04-10T02:26:02Z 2024-02-26T03:05:28Z   NONE      

I am testing out code that uses xarray to process netcdf files, in particular to join multiple netcdf files into one along shared dimensions. This was working well, except sometimes when saving the netcdf file the process would hang.

I was able to whittle it down to this simple example: https://github.com/jessicaaustin/xarray_netcdf_hanging_issue

This is the code snippet at the core of the example:

# If you set lock=False then this runs fine every time. # Setting lock=None causes it to intermittently hang on mfd.to_netcdf with xr.open_mfdataset(['dataset.nc'], combine='by_coords', lock=None) as mfd: p = os.path.join('tmp', 'xarray_{}.nc'.format(uuid.uuid4().hex)) print(f"Writing data to {p}") mfd.to_netcdf(p) print("complete")

If you run this once, it's typically fine. But run it over and over again in a loop, and it'll eventually hang on mfd.to_netcdf. However if I set lock=False then it runs fine every time.

I've seen this with the following combos:

  • xarray=0.14.1
  • dask=2.9.1
  • netcdf4=1.5.3

and

  • xarray=0.15.1
  • dask=2.14.0
  • netcdf4=1.5.3

And I've tried it with different netcdf files and different computers.

Versions

Output of `xr.show_versions()` INSTALLED VERSIONS ------------------ commit: None python: 3.7.6 | packaged by conda-forge | (default, Mar 23 2020, 23:03:20) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.15.0-20-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.5 libnetcdf: 4.7.4 xarray: 0.15.1 pandas: 1.0.3 numpy: 1.18.1 scipy: None netCDF4: 1.5.3 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.1.1.2 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: 2.14.0 distributed: 2.14.0 matplotlib: None cartopy: None seaborn: None numbagg: None setuptools: 46.1.3.post20200325 pip: 20.0.2 conda: None pytest: None IPython: None sphinx: None 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/3961/reactions",
    "total_count": 4,
    "+1": 4,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
     13221727 issue

Links from other tables