issue_comments

Given the recent turn in discussion here, might be worthwhile to share some thoughts on issue ( https://github.com/dask/dask/issues/2694 ).

That seems sensible to me. It would also be a good way to ensure that XArray operations adhere to all of the dask.array checks.

from dask.array.utils import assert_eq

our own assert_eq functions that both invoke the single-threaded scheduler, and also do a variety of other sanity checks like ensuring that the expected and computed dtypes and shapes are the same, that the keynames in graphs are sensible, etc..

What do you think about using that in our tests, i.e. from dask.utils import asset_eq as assert_dask_equals?

Or we could even check for a dask object in our own assert_equals and pass off to dask's?

In the dask library itself we solve this by creating our own assert_eq functions that both invoke the single-threaded scheduler, and also do a variety of other sanity checks like ensuring that the expected and computed dtypes and shapes are the same, that the keynames in graphs are sensible, etc..

OK, I figured it out. We set dask.set_options(get=dask.get) in xarray/tests/__init__.py, which is convenient for debugging (warning: changing global state!). Naturally, that means dask ignores the value of __dask_scheduler__ attribute.

I don't really have better ideas for how to test this, but it is reassuring that __dask_scheduler__ is the only attribute where we could reasonably expect to encounter this issue and its implementation is basically trivial. I suppose we could make one basic sanity test where we test each xarray data structure using the default scheduler. Possibly it would also make sense to change the scheduler explicitly (e.g., via a helper function for equality checks) rather than overloading it for all tests.

I can remove all the mock related code from test_dask.py entirely and test_dataarray_with_dask_coords still passes.

It was just a guess. Something wacky is certainly happening though. I recommend copying my code from my last comment and running pytest on it either in the root directory or in xarray/tests. I found that the outcome differed depending on location.

I can remove all the mock related code from test_dask.py entirely and test_dataarray_with_dask_coords still passes.

I still don't understand what actually went wrong here. It looks like we have test coverage for calling compute on an xarray.DataArray (see test_dataarray_with_dask_coords), but even though the exact example from that test fails at the repl the test passes when called with pytest:

I experienced some odd behavior when testing this within the XArray test suite

This file would pass when within xarray/tests/ but would fail when within the root directory

```python import numpy as np import xarray as xr import dask

def test_dask_reduction(): data = xr.DataArray(np.random.random(size=(10, 2)), dims=['samples', 'features']).chunk((5, 2)) result = dask.compute(data.mean(axis=0)) ```

I suspect some odd behavior around mock, but that's probably due to a general bias/lack of understanding of that module.

See https://github.com/pydata/xarray/pull/1760 for a potential fix

any suggestions on comprehensive ways to test things?

I still don't understand what actually went wrong here. It looks like we have test coverage for calling compute on an xarray.DataArray (see test_dataarray_with_dask_coords), but even though the exact example from that test fails at the repl the test passes when called with pytest: ``` In [5]: import xarray as xr

In [6]: import numpy as np

In [7]: import dask

In [8]: import dask.array as da

In [9]: import toolz ...: x = xr.Variable('x', da.arange(8, chunks=(4,))) ...: y = xr.Variable('y', da.arange(8, chunks=(4,)) * 2) ...: data = da.random.random((8, 8), chunks=(4, 4)) + 1 ...: array = xr.DataArray(data, dims=['x', 'y']) ...: array.coords['xx'] = x ...: array.coords['yy'] = y ...: ...: assert dict(array.dask_graph()) == toolz.merge(data.dask_graph(), ...: x.dask_graph(), ...: y.dask_graph()) ...: ...: (array2,) = dask.compute(array) ...:

ValueError Traceback (most recent call last) <ipython-input-9-4aaee405bed6> in <module>() 11 y.dask_graph()) 12 ---> 13 (array2,) = dask.compute(array)

~/conda/envs/xarray-py36/lib/python3.6/site-packages/dask/base.py in compute(args, kwargs) 334 results_iter = iter(results) 335 return tuple(a if f is None else f(next(results_iter), a) --> 336 for f, a in postcomputes) 337 338

~/conda/envs/xarray-py36/lib/python3.6/site-packages/dask/base.py in <genexpr>(.0) 334 results_iter = iter(results) 335 return tuple(a if f is None else f(next(results_iter), *a) --> 336 for f, a in postcomputes) 337 338

~/dev/xarray/xarray/core/dataarray.py in _dask_finalize(results, func, args, name) 607 @staticmethod 608 def _dask_finalize(results, func, args, name): --> 609 ds = func(results, *args) 610 variable = ds._variables.pop(_THIS_ARRAY) 611 coords = ds._variables

~/dev/xarray/xarray/core/dataset.py in _dask_postcompute(results, info, args) 551 func, args2 = v 552 r = results2.pop() --> 553 result = func(r, args2) 554 else: 555 result = v

~/dev/xarray/xarray/core/variable.py in _dask_finalize(results, array_func, array_args, dims, attrs, encoding) 389 results = {k: v for k, v in results.items() if k[0] == name} # cull 390 data = array_func(results, *array_args) --> 391 return Variable(dims, data, attrs=attrs, encoding=encoding) 392 393 @property

~/dev/xarray/xarray/core/variable.py in init(self, dims, data, attrs, encoding, fastpath) 267 """ 268 self._data = as_compatible_data(data, fastpath=fastpath) --> 269 self._dims = self._parse_dimensions(dims) 270 self._attrs = None 271 self._encoding = None

~/dev/xarray/xarray/core/variable.py in _parse_dimensions(self, dims) 431 raise ValueError('dimensions %s must have the same length as the ' 432 'number of data dimensions, ndim=%s' --> 433 % (dims, self.ndim)) 434 return dims 435

ValueError: dimensions ('x',) must have the same length as the number of data dimensions, ndim=0 ```

Also worth pointing out that this is likely the kind of bug that would have been caught with static typing

On Mon, Dec 4, 2017 at 6:55 PM, Stephan Hoyer notifications@github.com wrote:

Any objection to having the .compute methods point to dask.compute if the dask.version is appropriate?

Yes, this seems like a small win.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/1759#issuecomment-349148049, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszNQKcKZIvhT1rkNGONxU3rIH6gwOks5s9IZygaJpZM4Q1Xng .

Any objection to having the .compute methods point to dask.compute if the dask.version is appropriate?

Yes, this seems like a small win.

Any objection to having the .compute methods point to dask.compute if the dask.__version__ is appropriate?

Here is the problem. It was just a silly typo.

```diff diff --git a/xarray/core/dataarray.py b/xarray/core/dataarray.py index 0516b47..263860d 100644 --- a/xarray/core/dataarray.py +++ b/xarray/core/dataarray.py @@ -594,7 +594,7 @@ class DataArray(AbstractArray, BaseDataObject):

 @property
 def __dask_scheduler__(self):

• return self._to_temp_dataset().dask_optimize

• return self._to_temp_dataset().dask_scheduler

  def dask_postcompute(self): func, args = self._to_temp_dataset().dask_postcompute() ```

@shoyer any suggestions on comprehensive ways to test things?

While testing this I oddly learn that the following line makes this pass

python from xarray.tests import mock

I'll take a look

I don't think you're doing anything wrong. This looks like a bug related to @mrocklin's recent addition of dask duck methods for xarray.

Here are the values being passed to xarray.Dataset._dask_postcompute: results = ('mean_agg-aggregate-21a3ca7e382bdeee43ea06bee1ce3feb', 0) info = [(True, <this-array>, (<function Variable._dask_finalize at 0x118156158>, (<function finalize at 0x118350730>, (), ('y',), None, None)))] args = (set(), {'y': 2}, None, None, None)

It seems like something is going wrong on the dask side: results should be the computed numpy array, not an uncomputed dask key.