issue_comments

Hi all, I finally figured out my problem. On each independent process xr.open_mfdataset() seems to naturally try to do some multi-threaded access (even without parallel option ?). Each node of my cluster was configured in such a way that multi-threading was possible (my mistake). Here was my yaml config file used by PBSCluster() jobqueue: pbs: name: dask-worker # Dask worker options cores: 56 processes: 28 I tough that the parallel=True option was to enable parallelized access for my independent process. It actually enable parallelized access for possible threads of each process. Now, I have removed parallel=True from xr.open_mfdataset() call and ensure 1 thread by process by changing my config file: cores: 28 processes: 28 Thanks again for your help

@jhamman I was aware of the difference between the two parallel options. I was thus wondering if I could pass a parallel option to the netcdf4 library via the open_mfdataset() call. I tried to change the engine to netcdf4 and added the backend_kwarg : backend_kwargs={'parallel':True} but I get the same error. I 'll try the suggestion of Stephan to see how it behaves and I will report back. Thanks

To clear a few things up, the parallel option in netCDF4.Dataset is not the same as the parallel option in xarray.opne_mfdataset. In xarray, that option is meant to help speed up the time it takes to open many files at once. If you are using dask distributed, this should be done using that scheduler.

If you are only seeing thread parallelism in the open_mfdataset(..., parallel=True) call, I would start by looking at your dask distributed setup.

Can you try this workflow with and without the parallel option and report back:

Python client = Client(...) ds = xr.open_mfdataset(myfiles_path, concat_dim='t', engine='h5netcdf', paralel=...) x = ds['x'].load().data y = ds['y'].load().data ds.close()

Provided that you are setting up distributed to use multiple processes, you should get parallelism from multiple processes in this case.

This may be fixed if you try the development version of xarray -- we did a major refactor of xarray's handling of netCDF files, e.g., try pip install https://github.com/pydata/xarray/archive/master.zip.