issue_comments

Any final comments on this? If not, I'll probably merge this in the next day or two.

All the test are passing here. I would appreciate another round of reviews.

@shoyer - all of your previous comments have been addressed.

I've added some additional tests and cleaned up the implementation a bit. I'd like to get reviews from a few folks and hopefully get this merged later this week.

Thanks all for the comments. I will clean this up a bit and request a full review later this week.

A few things to note:

1. I have not tested save_mfdataset yet. In theory, it should work now but it will require some testing. I'll save that for another PR.
2. I will raise an informative error message when either h5netcdf or scipy are used to write files along with distributed.

The battle of inches continues. Turning off HDF5's file locking fixes all the tests for netCDF4 (🎉 ). Scipy is not working and h5netcdf doesn't support autoclose so it isn't expected to work.

@shoyer - I don't totally understand the scipy constraints on incremental writes but could that be playing a factor here?

I've this down to 4 test failures:

test_dask_distributed_netcdf_integration_test[NETCDF3_CLASSIC-True-scipy] test_dask_distributed_netcdf_integration_test[NETCDF3_CLASSIC-False-scipy] test_dask_distributed_netcdf_integration_test[NETCDF4_CLASSIC-False-netcdf4] test_dask_distributed_netcdf_integration_test[NETCDF4-False-netcdf4]

I think I'm ready for an initial review. I've made some changes to autoclose and sync so I'd like to get feedback on my approach before I spend too much time sorting out the last few failures.

This is mostly working now. I'm getting a test failure from open_dataset + distributed + autoclose so there is something to sort out there.

@shoyer - I have this working with the netcdf4 backend for with the NETCDF3_CLASSIC file format. I'm still having some locking issues with the HDF library and I'm not sure why.

I think we're getting close. We're currently failing during the sync step and I'm hypothesizing that it is due to the file not being closed after the setup steps. That said, I wasn't able to pinpoint why/where we're missing a close. I think this traceback is pretty informative:

Traceback (most recent call last): File "/Users/jhamman/anaconda/envs/xarray36/lib/python3.6/site-packages/distributed/worker.py", line 1255, in add_task self.tasks[key] = _deserialize(function, args, kwargs, task) File "/Users/jhamman/anaconda/envs/xarray36/lib/python3.6/site-packages/distributed/worker.py", line 641, in _deserialize args = pickle.loads(args) File "/Users/jhamman/anaconda/envs/xarray36/lib/python3.6/site-packages/distributed/protocol/pickle.py", line 59, in loads return pickle.loads(x) File "/Users/jhamman/Dropbox/src/xarray/xarray/backends/common.py", line 445, in __setstate__ self.ds = self._opener(mode=self._mode) File "/Users/jhamman/Dropbox/src/xarray/xarray/backends/netCDF4_.py", line 204, in _open_netcdf4_group ds = nc4.Dataset(filename, mode=mode, **kwargs) File "netCDF4/_netCDF4.pyx", line 2015, in netCDF4._netCDF4.Dataset.__init__ File "netCDF4/_netCDF4.pyx", line 1636, in netCDF4._netCDF4._ensure_nc_success OSError: [Errno -101] NetCDF: HDF error: b'/var/folders/v0/qnh7jvgx5gnglpxfztxdlhk00000gn/T/tmpn_mo662_/temp-0.nc' distributed.scheduler - ERROR - error from worker tcp://127.0.0.1:63248: [Errno -101] NetCDF: HDF error: b'/var/folders/v0/qnh7jvgx5gnglpxfztxdlhk00000gn/T/tmpn_mo662_/temp-0.nc' HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 140736093991744: #000: H5F.c line 586 in H5Fopen(): unable to open file major: File accessibilty minor: Unable to open file #001: H5Fint.c line 1305 in H5F_open(): unable to lock the file major: File accessibilty minor: Unable to open file #002: H5FD.c line 1839 in H5FD_lock(): driver lock request failed major: Virtual File Layer minor: Can't update object #003: H5FDsec2.c line 940 in H5FD_sec2_lock(): unable to lock file, errno = 35, error message = 'Resource temporarily unavailable' major: File accessibilty minor: Bad file ID accessed

@mrocklin - What is the preferred method for determining which scheduler is being used?

The tests failure indicates that the netcdf4/h5netcdf libraries cannot open the file in write/append mode, and it seems that is because the file is already open (by another process).

Two questions:

1. autoclose is False to_netcdf. That generally makes sense to me but I'm concerned that we're not being explicit enough about closing the file after each process is done interacting with it. Do we have a way to lock until the file is closed?
2. The lock we're using is dask's SerializableLock. Is that the correct Lock to be using? There is also the distributed.Lock.

xref: https://github.com/dask/dask/issues/1892

Thanks @mrocklin for taking a look here. I reworked the tests a bit more to put the to_netcdf inside the distributed cluster section.

Bad news is that the tests are failing again. The good news is we have a semi-informative error message that indicates we're missing a lock somewhere.

Link to most descriptive failing test: https://travis-ci.org/pydata/xarray/jobs/336643000#L5076

xref: https://github.com/pydata/xarray/issues/798 and https://github.com/dask/dask/issues/2488 which are both seem to be relevant to this discussion.

I'm also remembering @pwolfram was quite involved with the original distributed integration so pinging him to see if he is interested in this.

Yes, the zarr backend here in xarray is also using dask.array.store and seems to work with distributed just fine.

I've just taken another swing at this and come up empty. I open to ideas in the following areas:

1. scipy backend is failing to roundtrip a length 1 datetime array: https://travis-ci.org/pydata/xarray/jobs/333068098#L4504
2. scipy, netcdf4, and h5netcdf backends are all failing inside dask-distributed: https://travis-ci.org/pydata/xarray/jobs/333068098#L4919

The good news here is that only 8 tests are failing after applying the array wrapper so I suspect we're quite close. I'm hoping @shoyer may have some ideas on (1) since I think he had implemented some scipy workarounds in the past. @mrocklin, I'm hoping you can point me in the right direction.

All of these tests are reproducible locally.

(BTW, I have a use case that is going to need this functionality so I'm personally motivated to see it across the finish line)

@mrocklin -

I have a test failing here with a familiar message.

E       TypeError: 'Future' object is not iterable

We saw this last week when debugging some pangeo things. Can you remind me what our solution was?