html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue https://github.com/pydata/xarray/issues/1759#issuecomment-349772394,https://api.github.com/repos/pydata/xarray/issues/1759,349772394,MDEyOklzc3VlQ29tbWVudDM0OTc3MjM5NA==,3019665,2017-12-06T20:57:51Z,2017-12-06T20:57:51Z,NONE,"Given the recent turn in discussion here, might be worthwhile to share some thoughts on issue ( https://github.com/dask/dask/issues/2694 ).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,279161550 https://github.com/pydata/xarray/issues/1759#issuecomment-349765925,https://api.github.com/repos/pydata/xarray/issues/1759,349765925,MDEyOklzc3VlQ29tbWVudDM0OTc2NTkyNQ==,306380,2017-12-06T20:32:58Z,2017-12-06T20:32:58Z,MEMBER,"That seems sensible to me. It would also be a good way to ensure that XArray operations adhere to all of the dask.array checks. from dask.array.utils import assert_eq","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,279161550 https://github.com/pydata/xarray/issues/1759#issuecomment-349763381,https://api.github.com/repos/pydata/xarray/issues/1759,349763381,MDEyOklzc3VlQ29tbWVudDM0OTc2MzM4MQ==,5635139,2017-12-06T20:24:08Z,2017-12-06T20:24:08Z,MEMBER,"> our own assert_eq functions that both invoke the single-threaded scheduler, and also do a variety of other sanity checks like ensuring that the expected and computed dtypes and shapes are the same, that the keynames in graphs are sensible, etc.. What do you think about using that in our tests, i.e. `from dask.utils import asset_eq as assert_dask_equals`? Or we could even check for a dask object in our own `assert_equals` and pass off to dask's?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,279161550 https://github.com/pydata/xarray/issues/1759#issuecomment-349630978,https://api.github.com/repos/pydata/xarray/issues/1759,349630978,MDEyOklzc3VlQ29tbWVudDM0OTYzMDk3OA==,306380,2017-12-06T12:52:56Z,2017-12-06T12:52:56Z,MEMBER,"In the dask library itself we solve this by creating our own `assert_eq` functions that both invoke the single-threaded scheduler, and also do a variety of other sanity checks like ensuring that the expected and computed dtypes and shapes are the same, that the keynames in graphs are sensible, etc.. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,279161550 https://github.com/pydata/xarray/issues/1759#issuecomment-349537178,https://api.github.com/repos/pydata/xarray/issues/1759,349537178,MDEyOklzc3VlQ29tbWVudDM0OTUzNzE3OA==,1217238,2017-12-06T05:20:29Z,2017-12-06T05:20:29Z,MEMBER,"OK, I figured it out. We set `dask.set_options(get=dask.get)` in `xarray/tests/__init__.py`, which is convenient for debugging (warning: changing global state!). Naturally, that means dask ignores the value of `__dask_scheduler__` attribute. I don't really have better ideas for how to test this, but it is reassuring that `__dask_scheduler__` is the only attribute where we could reasonably expect to encounter this issue and its implementation is basically trivial. I suppose we could make one basic sanity test where we test each xarray data structure using the default scheduler. Possibly it would also make sense to change the scheduler explicitly (e.g., via a helper function for equality checks) rather than overloading it for all tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,279161550 https://github.com/pydata/xarray/issues/1759#issuecomment-349452364,https://api.github.com/repos/pydata/xarray/issues/1759,349452364,MDEyOklzc3VlQ29tbWVudDM0OTQ1MjM2NA==,306380,2017-12-05T21:47:18Z,2017-12-05T21:47:18Z,MEMBER,"> I can remove all the mock related code from test_dask.py entirely and test_dataarray_with_dask_coords still passes. It was just a guess. Something wacky is certainly happening though. I recommend copying my code from my last comment and running pytest on it either in the root directory or in `xarray/tests`. I found that the outcome differed depending on location.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,279161550 https://github.com/pydata/xarray/issues/1759#issuecomment-349391827,https://api.github.com/repos/pydata/xarray/issues/1759,349391827,MDEyOklzc3VlQ29tbWVudDM0OTM5MTgyNw==,1217238,2017-12-05T18:13:18Z,2017-12-05T18:13:18Z,MEMBER,I can remove all the mock related code from `test_dask.py` entirely and `test_dataarray_with_dask_coords` still passes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,279161550 https://github.com/pydata/xarray/issues/1759#issuecomment-349151434,https://api.github.com/repos/pydata/xarray/issues/1759,349151434,MDEyOklzc3VlQ29tbWVudDM0OTE1MTQzNA==,306380,2017-12-05T00:14:13Z,2017-12-05T00:14:13Z,MEMBER,"> I still don't understand what actually went wrong here. It looks like we have test coverage for calling compute on an xarray.DataArray (see test_dataarray_with_dask_coords), but even though the exact example from that test fails at the repl the test passes when called with pytest: I experienced some odd behavior when testing this within the XArray test suite This file would pass when within `xarray/tests/` but would fail when within the root directory ```python import numpy as np import xarray as xr import dask def test_dask_reduction(): data = xr.DataArray(np.random.random(size=(10, 2)), dims=['samples', 'features']).chunk((5, 2)) result = dask.compute(data.mean(axis=0)) ``` I suspect some odd behavior around mock, but that's probably due to a general bias/lack of understanding of that module.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,279161550 https://github.com/pydata/xarray/issues/1759#issuecomment-349151169,https://api.github.com/repos/pydata/xarray/issues/1759,349151169,MDEyOklzc3VlQ29tbWVudDM0OTE1MTE2OQ==,306380,2017-12-05T00:12:38Z,2017-12-05T00:12:38Z,MEMBER,"See https://github.com/pydata/xarray/pull/1760 for a potential fix ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,279161550 https://github.com/pydata/xarray/issues/1759#issuecomment-349149824,https://api.github.com/repos/pydata/xarray/issues/1759,349149824,MDEyOklzc3VlQ29tbWVudDM0OTE0OTgyNA==,1217238,2017-12-05T00:04:46Z,2017-12-05T00:04:46Z,MEMBER,"> any suggestions on comprehensive ways to test things? I still don't understand what actually went wrong here. It looks like we have test coverage for calling compute on an xarray.DataArray (see `test_dataarray_with_dask_coords`), but even though the exact example from that test fails at the repl the test passes when called with pytest: ``` In [5]: import xarray as xr In [6]: import numpy as np In [7]: import dask In [8]: import dask.array as da In [9]: import toolz ...: x = xr.Variable('x', da.arange(8, chunks=(4,))) ...: y = xr.Variable('y', da.arange(8, chunks=(4,)) * 2) ...: data = da.random.random((8, 8), chunks=(4, 4)) + 1 ...: array = xr.DataArray(data, dims=['x', 'y']) ...: array.coords['xx'] = x ...: array.coords['yy'] = y ...: ...: assert dict(array.__dask_graph__()) == toolz.merge(data.__dask_graph__(), ...: x.__dask_graph__(), ...: y.__dask_graph__()) ...: ...: (array2,) = dask.compute(array) ...: --------------------------------------------------------------------------- ValueError Traceback (most recent call last) in () 11 y.__dask_graph__()) 12 ---> 13 (array2,) = dask.compute(array) ~/conda/envs/xarray-py36/lib/python3.6/site-packages/dask/base.py in compute(*args, **kwargs) 334 results_iter = iter(results) 335 return tuple(a if f is None else f(next(results_iter), *a) --> 336 for f, a in postcomputes) 337 338 ~/conda/envs/xarray-py36/lib/python3.6/site-packages/dask/base.py in (.0) 334 results_iter = iter(results) 335 return tuple(a if f is None else f(next(results_iter), *a) --> 336 for f, a in postcomputes) 337 338 ~/dev/xarray/xarray/core/dataarray.py in _dask_finalize(results, func, args, name) 607 @staticmethod 608 def _dask_finalize(results, func, args, name): --> 609 ds = func(results, *args) 610 variable = ds._variables.pop(_THIS_ARRAY) 611 coords = ds._variables ~/dev/xarray/xarray/core/dataset.py in _dask_postcompute(results, info, *args) 551 func, args2 = v 552 r = results2.pop() --> 553 result = func(r, *args2) 554 else: 555 result = v ~/dev/xarray/xarray/core/variable.py in _dask_finalize(results, array_func, array_args, dims, attrs, encoding) 389 results = {k: v for k, v in results.items() if k[0] == name} # cull 390 data = array_func(results, *array_args) --> 391 return Variable(dims, data, attrs=attrs, encoding=encoding) 392 393 @property ~/dev/xarray/xarray/core/variable.py in __init__(self, dims, data, attrs, encoding, fastpath) 267 """""" 268 self._data = as_compatible_data(data, fastpath=fastpath) --> 269 self._dims = self._parse_dimensions(dims) 270 self._attrs = None 271 self._encoding = None ~/dev/xarray/xarray/core/variable.py in _parse_dimensions(self, dims) 431 raise ValueError('dimensions %s must have the same length as the ' 432 'number of data dimensions, ndim=%s' --> 433 % (dims, self.ndim)) 434 return dims 435 ValueError: dimensions ('x',) must have the same length as the number of data dimensions, ndim=0 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,279161550 https://github.com/pydata/xarray/issues/1759#issuecomment-349149217,https://api.github.com/repos/pydata/xarray/issues/1759,349149217,MDEyOklzc3VlQ29tbWVudDM0OTE0OTIxNw==,306380,2017-12-05T00:01:20Z,2017-12-05T00:01:20Z,MEMBER,"Also worth pointing out that this is likely the kind of bug that would have been caught with static typing On Mon, Dec 4, 2017 at 6:55 PM, Stephan Hoyer wrote: > Any objection to having the .compute methods point to dask.compute if the > dask.*version* is appropriate? > > Yes, this seems like a small win. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > , or mute > the thread > > . > ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,279161550 https://github.com/pydata/xarray/issues/1759#issuecomment-349148049,https://api.github.com/repos/pydata/xarray/issues/1759,349148049,MDEyOklzc3VlQ29tbWVudDM0OTE0ODA0OQ==,1217238,2017-12-04T23:55:28Z,2017-12-04T23:55:28Z,MEMBER,"> Any objection to having the .compute methods point to dask.compute if the dask.__version__ is appropriate? Yes, this seems like a small win.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,279161550 https://github.com/pydata/xarray/issues/1759#issuecomment-349145345,https://api.github.com/repos/pydata/xarray/issues/1759,349145345,MDEyOklzc3VlQ29tbWVudDM0OTE0NTM0NQ==,306380,2017-12-04T23:43:05Z,2017-12-04T23:43:05Z,MEMBER,Any objection to having the `.compute` methods point to `dask.compute` if the `dask.__version__` is appropriate?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,279161550 https://github.com/pydata/xarray/issues/1759#issuecomment-349144660,https://api.github.com/repos/pydata/xarray/issues/1759,349144660,MDEyOklzc3VlQ29tbWVudDM0OTE0NDY2MA==,306380,2017-12-04T23:40:18Z,2017-12-04T23:40:18Z,MEMBER,"Here is the problem. It was just a silly typo. ```diff diff --git a/xarray/core/dataarray.py b/xarray/core/dataarray.py index 0516b47..263860d 100644 --- a/xarray/core/dataarray.py +++ b/xarray/core/dataarray.py @@ -594,7 +594,7 @@ class DataArray(AbstractArray, BaseDataObject): @property def __dask_scheduler__(self): - return self._to_temp_dataset().__dask_optimize__ + return self._to_temp_dataset().__dask_scheduler__ def __dask_postcompute__(self): func, args = self._to_temp_dataset().__dask_postcompute__() ``` @shoyer any suggestions on comprehensive ways to test things?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,279161550 https://github.com/pydata/xarray/issues/1759#issuecomment-349141548,https://api.github.com/repos/pydata/xarray/issues/1759,349141548,MDEyOklzc3VlQ29tbWVudDM0OTE0MTU0OA==,306380,2017-12-04T23:25:43Z,2017-12-04T23:25:43Z,MEMBER,"While testing this I oddly learn that the following line makes this pass ```python from xarray.tests import mock ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,279161550 https://github.com/pydata/xarray/issues/1759#issuecomment-349139597,https://api.github.com/repos/pydata/xarray/issues/1759,349139597,MDEyOklzc3VlQ29tbWVudDM0OTEzOTU5Nw==,306380,2017-12-04T23:17:47Z,2017-12-04T23:17:47Z,MEMBER,I'll take a look,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,279161550 https://github.com/pydata/xarray/issues/1759#issuecomment-349134458,https://api.github.com/repos/pydata/xarray/issues/1759,349134458,MDEyOklzc3VlQ29tbWVudDM0OTEzNDQ1OA==,1217238,2017-12-04T22:54:47Z,2017-12-04T22:54:47Z,MEMBER,"I don't think you're doing anything wrong. This looks like a bug related to @mrocklin's recent addition of dask duck methods for xarray. Here are the values being passed to `xarray.Dataset._dask_postcompute`: ``` results = ('mean_agg-aggregate-21a3ca7e382bdeee43ea06bee1ce3feb', 0) info = [(True, , (, (, (), ('y',), None, None)))] args = (set(), {'y': 2}, None, None, None) ``` It seems like something is going wrong on the dask side: `results` should be the computed numpy array, not an uncomputed dask key.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,279161550