html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/pull/3847#issuecomment-643496930,https://api.github.com/repos/pydata/xarray/issues/3847,643496930,MDEyOklzc3VlQ29tbWVudDY0MzQ5NjkzMA==,14808389,2020-06-12T21:49:47Z,2020-06-12T21:50:02Z,MEMBER,"I think we could bump the version even further (at least to `2.8`, maybe also `2.9`) so that should probably be a different PR","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-643480505,https://api.github.com/repos/pydata/xarray/issues/3847,643480505,MDEyOklzc3VlQ29tbWVudDY0MzQ4MDUwNQ==,10194086,2020-06-12T20:58:30Z,2020-06-12T20:58:30Z,MEMBER,Do you need to add a note to what's new that the dask version is bumped?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-643448649,https://api.github.com/repos/pydata/xarray/issues/3847,643448649,MDEyOklzc3VlQ29tbWVudDY0MzQ0ODY0OQ==,14808389,2020-06-12T19:31:37Z,2020-06-12T19:31:37Z,MEMBER,unless there are any objections (@shoyer?) I'm going to merge this tomorrow.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-636833504,https://api.github.com/repos/pydata/xarray/issues/3847,636833504,MDEyOklzc3VlQ29tbWVudDYzNjgzMzUwNA==,14808389,2020-06-01T12:31:20Z,2020-06-01T12:31:20Z,MEMBER,"gentle ping, @shoyer","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-634888639,https://api.github.com/repos/pydata/xarray/issues/3847,634888639,MDEyOklzc3VlQ29tbWVudDYzNDg4ODYzOQ==,14808389,2020-05-27T19:23:30Z,2020-05-27T19:24:14Z,MEMBER,"anyway, with this the tests finally pass :tada: so this should be ready for review and merging.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-634876945,https://api.github.com/repos/pydata/xarray/issues/3847,634876945,MDEyOklzc3VlQ29tbWVudDYzNDg3Njk0NQ==,14808389,2020-05-27T19:04:40Z,2020-05-27T19:04:40Z,MEMBER,"then it seems that this PR removes this intentional restriction. I'm not sure it is still needed, though: we don't use `np.testing.assert_allclose` internally because we want to provide our own error messages, and the actual comparison is done with https://github.com/pydata/xarray/blob/e5cc19cd8f8a69e0743f230f5bf51b7778a0ff96/xarray/core/duck_array_ops.py#L204
which should allow dispatching","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-634868914,https://api.github.com/repos/pydata/xarray/issues/3847,634868914,MDEyOklzc3VlQ29tbWVudDYzNDg2ODkxNA==,1217238,2020-05-27T18:50:12Z,2020-05-27T18:50:12Z,MEMBER,"> I'm confused, I thought that `xr.testing.assert_allclose` explicitly supported duck arrays (it calls `duck_array_ops.allclose_or_equiv`).
Right, it explicitly supports duck arrays -- by always converting them into NumPy!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-634867837,https://api.github.com/repos/pydata/xarray/issues/3847,634867837,MDEyOklzc3VlQ29tbWVudDYzNDg2NzgzNw==,14808389,2020-05-27T18:48:15Z,2020-05-27T18:48:15Z,MEMBER,"hmm... using `arr.compute()` fails while `np.array(arr)` works. For now, I'm converting all dask arrays to numpy if the dask version is not high enough (I can't do that only for `bool_` arrays since they have been converted to float somewhere).
@shoyer: I'm confused, I thought that `xr.testing.assert_allclose` explicitly supported duck arrays (it calls `duck_array_ops.allclose_or_equiv`). TBC, what I was talking about was these lines:
https://github.com/pydata/xarray/blob/e5cc19cd8f8a69e0743f230f5bf51b7778a0ff96/xarray/testing.py#L124
https://github.com/pydata/xarray/blob/e5cc19cd8f8a69e0743f230f5bf51b7778a0ff96/xarray/testing.py#L132-L135
where the `.values` always convert to `numpy`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-634855041,https://api.github.com/repos/pydata/xarray/issues/3847,634855041,MDEyOklzc3VlQ29tbWVudDYzNDg1NTA0MQ==,1217238,2020-05-27T18:25:40Z,2020-05-27T18:25:40Z,MEMBER,"> So it seems that `assert_allclose` never worked with `dask` but always converted to `numpy`. Would it make sense to investigate further, or is it better to just wait until we can merge this?
This was intentional, I think. `np.testing.assert_allclose()` doesn't support dispatching -- it always converts into NumPy arrays. I'm not sure that `assert_allclose` would even be well defined on dask arrays otherwise, because it doesn't have any output (it only raises an error).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-634826398,https://api.github.com/repos/pydata/xarray/issues/3847,634826398,MDEyOklzc3VlQ29tbWVudDYzNDgyNjM5OA==,14808389,2020-05-27T17:39:51Z,2020-05-27T17:39:51Z,MEMBER,let's see if that works,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-634811793,https://api.github.com/repos/pydata/xarray/issues/3847,634811793,MDEyOklzc3VlQ29tbWVudDYzNDgxMTc5Mw==,2448579,2020-05-27T17:14:17Z,2020-05-27T17:14:17Z,MEMBER,"How about we call `compute` in `assert_allclose` for boolean dask arrays when `dask < 2.9.1`? This bit can then be removed in a couple of months.
This compute was happening in previous versions anyway (because `assert_allclose` was using `.values`) so this would not be a regression.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-634611339,https://api.github.com/repos/pydata/xarray/issues/3847,634611339,MDEyOklzc3VlQ29tbWVudDYzNDYxMTMzOQ==,14808389,2020-05-27T11:57:18Z,2020-05-27T11:57:18Z,MEMBER,"I needed to use `assert_allclose` in the `pint` tests in #3975, so I modified it to use `.data` instead of `.values` and now the exact same tests are failing. So it seems that `assert_allclose` never worked with `dask` but always converted to `numpy`. Would it make sense to investigate further, or is it better to just wait until we can merge this?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-628198836,https://api.github.com/repos/pydata/xarray/issues/3847,628198836,MDEyOklzc3VlQ29tbWVudDYyODE5ODgzNg==,2448579,2020-05-13T19:28:42Z,2020-05-13T19:28:42Z,MEMBER,"> Would it make sense to wait until we can bump the dask version to 2.9?
Sounds good to me. It's only 1.5 months away","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-628185474,https://api.github.com/repos/pydata/xarray/issues/3847,628185474,MDEyOklzc3VlQ29tbWVudDYyODE4NTQ3NA==,14808389,2020-05-13T19:03:14Z,2020-05-13T19:03:14Z,MEMBER,"Unfortunately, simply replacing `assert np.allclose(...)` with `np.testing.assert_allclose(...)` does not fix this. Also, I undid the swapping of arguments to `allclose_or_equiv` which makes the `min-all-deps` CI fail, too. Does anyone know why that works?
Would it make sense to wait until we can bump the dask version to `2.9`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-628090382,https://api.github.com/repos/pydata/xarray/issues/3847,628090382,MDEyOklzc3VlQ29tbWVudDYyODA5MDM4Mg==,2448579,2020-05-13T16:07:03Z,2020-05-13T16:07:03Z,MEMBER,this comment: `# Numpy < 1.13 does not handle object-type array.` suggests that this code needs to be updated for our new `numpy>=1.15` requirement,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-628083718,https://api.github.com/repos/pydata/xarray/issues/3847,628083718,MDEyOklzc3VlQ29tbWVudDYyODA4MzcxOA==,1217238,2020-05-13T15:55:52Z,2020-05-13T15:55:52Z,MEMBER,"It looks like there might be some sort of issue with dask's numpy compatibility layer? e.g., in np.allclose?
These fallbacks are catching quite a few errors....
One idea would be to try plumbing this logic into `np.testing.assert_allclose` (which doesn't do dispatching) rather than `np.allclose`. The former comes pre-packaged with better error messages, which might be handy.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-628076804,https://api.github.com/repos/pydata/xarray/issues/3847,628076804,MDEyOklzc3VlQ29tbWVudDYyODA3NjgwNA==,2448579,2020-05-13T15:44:10Z,2020-05-13T15:44:10Z,MEMBER,"example failure:
```
___________________ test_reduce[None-True-var-True-bool_-1] ____________________
dim_num = 1, dtype = , dask = True, func = 'var'
skipna = True, aggdim = None
@pytest.mark.parametrize(""dim_num"", [1, 2])
@pytest.mark.parametrize(""dtype"", [float, int, np.float32, np.bool_])
@pytest.mark.parametrize(""dask"", [False, True])
@pytest.mark.parametrize(""func"", [""sum"", ""min"", ""max"", ""mean"", ""var""])
# TODO test cumsum, cumprod
@pytest.mark.parametrize(""skipna"", [False, True])
@pytest.mark.parametrize(""aggdim"", [None, ""x""])
def test_reduce(dim_num, dtype, dask, func, skipna, aggdim):
if aggdim == ""y"" and dim_num < 2:
pytest.skip(""dim not in this test"")
if dtype == np.bool_ and func == ""mean"":
pytest.skip(""numpy does not support this"")
if dask and not has_dask:
pytest.skip(""requires dask"")
if dask and skipna is False and dtype in [np.bool_]:
pytest.skip(""dask does not compute object-typed array"")
rtol = 1e-04 if dtype == np.float32 else 1e-05
da = construct_dataarray(dim_num, dtype, contains_nan=True, dask=dask)
axis = None if aggdim is None else da.get_axis_num(aggdim)
# TODO: remove these after resolving
# https://github.com/dask/dask/issues/3245
with warnings.catch_warnings():
warnings.filterwarnings(""ignore"", ""Mean of empty slice"")
warnings.filterwarnings(""ignore"", ""All-NaN slice"")
warnings.filterwarnings(""ignore"", ""invalid value encountered in"")
if da.dtype.kind == ""O"" and skipna:
# Numpy < 1.13 does not handle object-type array.
try:
if skipna:
expected = getattr(np, f""nan{func}"")(da.values, axis=axis)
else:
expected = getattr(np, func)(da.values, axis=axis)
actual = getattr(da, func)(skipna=skipna, dim=aggdim)
assert_dask_array(actual, dask)
assert np.allclose(
actual.values, np.array(expected), rtol=1.0e-4, equal_nan=True
)
except (TypeError, AttributeError, ZeroDivisionError):
# TODO currently, numpy does not support some methods such as
""casting='same_kind'""
> % (funcname(function), str(dtype), str(result.dtype))
)
E ValueError: Inferred dtype from function 'sub' was 'int64' but got 'float64', which can't be cast using casting='same_kind'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-625885180,https://api.github.com/repos/pydata/xarray/issues/3847,625885180,MDEyOklzc3VlQ29tbWVudDYyNTg4NTE4MA==,2448579,2020-05-08T15:58:59Z,2020-05-13T15:36:41Z,MEMBER,"I'm confused here. What did you change to trigger this error?
ALL WRONG: ~We could call `arr1.compute(), arr2.compute()` in
https://github.com/pydata/xarray/blob/69548df9826cde9df6cbdae9c033c9fb1e62d493/xarray/core/duck_array_ops.py#L203-L204~
~At that point, we need to compare actual values. I'm not sure that dask exits `all()` early if one chunk evaluates to False ( in which case this change would be a performance regression)~
EDIT: ignore this. We can't compute because it might blow memory.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-625554011,https://api.github.com/repos/pydata/xarray/issues/3847,625554011,MDEyOklzc3VlQ29tbWVudDYyNTU1NDAxMQ==,14808389,2020-05-07T23:52:27Z,2020-05-08T00:39:31Z,MEMBER,"@pydata/xarray, this is really close but I'm not familiar enough with `dask` to get it to work with `dask<2.9.1`.
Once we get the `py36-min-all-deps` and `py36-min-nep18` CI to pass, this should be ready for a final review and merging.
Edit: there seem to be a few failing tests related to `pint` (I will fix those) but the tests in question are in `test_duck_array_ops.py` and only fail with `py36-min-nep18`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-604749478,https://api.github.com/repos/pydata/xarray/issues/3847,604749478,MDEyOklzc3VlQ29tbWVudDYwNDc0OTQ3OA==,14808389,2020-03-27T00:10:27Z,2020-03-27T00:10:27Z,MEMBER,"The issue is with how `dask<2.9.1` handles dtypes on compute (in `nanvar`?) when the data is an array with dtype `object` filled with `False` and some missing values represented by `np.nan`.
I really lack experience with `dask`, though, so I'm clueless as to what to do to fix that (besides calling `compute` before passing to `allclose_or_equiv`) and would appreciate help with this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-601350745,https://api.github.com/repos/pydata/xarray/issues/3847,601350745,MDEyOklzc3VlQ29tbWVudDYwMTM1MDc0NQ==,14808389,2020-03-19T18:40:41Z,2020-03-19T18:40:41Z,MEMBER,"unfortunately, we need `dask>=2.9.1` for that. I could try debugging a bit more to find out exactly why it fails (or someone helps me with that?), but that would take me a bit more time than just skipping / xfailing those tests if on `dask<2.9.1`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-601321231,https://api.github.com/repos/pydata/xarray/issues/3847,601321231,MDEyOklzc3VlQ29tbWVudDYwMTMyMTIzMQ==,2448579,2020-03-19T17:43:36Z,2020-03-19T17:43:36Z,MEMBER,"Our minimum versions policy allows us to bump dask. So let's do that?
```
dask 2.2 (2019-08-01) 2.5 (2019-09-27) <
distributed 2.2 (2019-08-01) 2.5 (2019-09-27) <
```
> Also, where should I put the whats-new.rst entry?
New features? It is public API and is a very nice enhancement.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-601288023,https://api.github.com/repos/pydata/xarray/issues/3847,601288023,MDEyOklzc3VlQ29tbWVudDYwMTI4ODAyMw==,14808389,2020-03-19T16:46:49Z,2020-03-19T16:49:31Z,MEMBER,"to me the failing tests on the `min-all-deps` and `min-nep18` CI seem like a `dask` issue (in `var`?) that has been resolved in later versions: if we compute `actual` before passing it to `assert_allclose` the tests don't fail. How should I resolve this?
Also, where should I put the `whats-new.rst` entry?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-596516918,https://api.github.com/repos/pydata/xarray/issues/3847,596516918,MDEyOklzc3VlQ29tbWVudDU5NjUxNjkxOA==,14808389,2020-03-09T13:15:24Z,2020-03-09T13:15:24Z,MEMBER,now only the `min-all-deps` and `min-nep18` CI fail. I suspect that is due to the pinned dependencies.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-596374204,https://api.github.com/repos/pydata/xarray/issues/3847,596374204,MDEyOklzc3VlQ29tbWVudDU5NjM3NDIwNA==,2448579,2020-03-09T07:39:31Z,2020-03-09T07:39:31Z,MEMBER,Let's mark those as xfail then?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-596256394,https://api.github.com/repos/pydata/xarray/issues/3847,596256394,MDEyOklzc3VlQ29tbWVudDU5NjI1NjM5NA==,14808389,2020-03-08T21:44:30Z,2020-03-08T21:44:30Z,MEMBER,the failures should be a bug in `pint` and hopefully fixed by hgrecco/pint#1044,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749
https://github.com/pydata/xarray/pull/3847#issuecomment-596214732,https://api.github.com/repos/pydata/xarray/issues/3847,596214732,MDEyOklzc3VlQ29tbWVudDU5NjIxNDczMg==,5635139,2020-03-08T14:59:19Z,2020-03-08T14:59:19Z,MEMBER,Looks great!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,577425749