issue_comments

Closing this now. The distributed integration test module seems to be covering our IO use cases well enough. I don't think we need to do anything here at this time.

Seems like the distributed scheduler is the advised one to use in general, so maybe some tests could be added for this one. For sure for diskIO, would be interesting to see the difference.

http://dask.pydata.org/en/latest/setup.html

Note that the newer dask.distributed scheduler is often preferable even on single workstations. It contains many diagnostics and features not found in the older single-machine scheduler. The following pages explain in more detail how to set up Dask on a variety of local and distributed hardware.

FWIW most of the logic within the dask collections (array, dataframe, delayed) is only tested with dask.local.get_sync. This also makes the test suite much faster.

Obviously though for things like writing to disk it's useful to check different schedulers.

I managed to dig up some more information here. I was having a test failure in test_serializable_locks resulting in a traceback that looks like. ``` ... timeout_handle = self.add_timeout(self.time() + timeout, self.stop) self.start() if timeout is not None: self.remove_timeout(timeout_handle) if not future_cell[0].done():

      raise TimeoutError('Operation timed out after %s seconds' % timeout)

E tornado.ioloop.TimeoutError: Operation timed out after 10 seconds

../../../anaconda/envs/xarray36/lib/python3.6/site-packages/tornado/ioloop.py:457: TimeoutError ```

From then on we were using the distributed scheduler and any tests that used dask resulted in a additional timeout (or similar error).

Unfortunately, my attempts to provide a mcve have come up short. If I can come up with one, I'll report upstream but as it is, I can't really transfer this behavior outside of my example.

cc @mrocklin

Huh, that's interesting. Yes, I suppose should at least consider parametric tests using both dask's multithreaded and distributed schedulers. Though I'll note that for test we actually set the default scheduler to dask's basic non-parallelized get, for easier debugging: https://github.com/pydata/xarray/blob/54468e1924174a03e7ead3be8545f687f084f4dd/xarray/tests/init.py#L87

For #1793, the key thing would be to ensure that we run the tests in the isolated context without changing the default scheduler.