issue_comments

I added a PR in order to allow a dev version of bottleneck to be used, too: https://github.com/pydata/xarray/issues/1235

@fmaussion thanks for puzzling this one out!

@ghisvail thanks for the report!

the tests now pass with bottleneck master.

@shoyer nevermind, I found the bug: https://github.com/kwgoodman/bottleneck/issues/161

Quite a tricky one indeed ;-)

@shoyer there is something very weird going on. See the following example:

```python import numpy as np import bottleneck as bn import xarray as xr

da = xr.DataArray(np.ones((10, 20)).astype(np.int), dims=['x', 'y'], coords={'abc':('y', np.array(['a'] * 9 + ['c'] + ['b'] * 10))}) np.testing.assert_allclose(np.sum(da[:, 9:10]), bn.nansum(da[:, 9:10])) # this will allways work np.testing.assert_allclose(da.groupby('abc').reduce(np.sum), da.groupby('abc').sum()) # this won't ` This will work with bottleneck installed. Now change theastype(np.int)toastype(np.float)`` and the first assertion would still pass, but the second won't! It will fail with:

```

AssertionError Traceback (most recent call last) <ipython-input-95-d34b19396eb5> in <module>() 2 coords={'abc':('y', np.array(['a'] * 9 + ['c'] + ['b'] * 10))}) 3 np.testing.assert_allclose(np.sum(da[:, 9:10]), bn.nansum(da[:, 9:10])) ----> 4 np.testing.assert_allclose(da.groupby('abc').reduce(np.sum), da.groupby('abc').sum())

/home/mowglie/.pyvirtualenvs/py3/lib/python3.4/site-packages/numpy/testing/utils.py in assert_allclose(actual, desired, rtol, atol, equal_nan, err_msg, verbose) 1409 header = 'Not equal to tolerance rtol=%g, atol=%g' % (rtol, atol) 1410 assert_array_compare(compare, actual, desired, err_msg=str(err_msg), -> 1411 verbose=verbose, header=header, equal_nan=equal_nan) 1412 1413

/home/mowglie/.pyvirtualenvs/py3/lib/python3.4/site-packages/numpy/testing/utils.py in assert_array_compare(comparison, x, y, err_msg, verbose, header, precision, equal_nan) 794 names=('x', 'y'), precision=precision) 795 if not cond: --> 796 raise AssertionError(msg) 797 except ValueError: 798 import traceback

AssertionError: Not equal to tolerance rtol=1e-07, atol=0

(mismatch 33.33333333333333%) x: array([ 90., 100., 10.]) y: array([ 90., 100., 1.]) ``` So the new operation is applied only to the first element of the grouped selection, and I didn't manage to us bottleneck only to replicate this (since the first assertion always passes).

Before I dig into this I'd like to have your opinion on this: any idea on what could go wrong here?

@ghisvail Thanks for your diligence on this.

@fmaussion If you can turn one of these into a test case for bottleneck to report upstream that would be super helpful. I would probably start with test_groupby_sum. It's likely that this only occurs for arrays with a particular strides (memory layout) and shape, which is where my blind guess that I suggested on the bottleneck tracker was inspired by.

I can confirm that - I have the same problems on my pip virtualenv on linux mint (I tried to reproduce it on travis without success)

OK, thanks for looking into this!

On Wed, Jan 18, 2017 at 10:36 AM, Ghislain Antony Vaillant notifications@github.com wrote:

We'd need to wait for numpy-1.12.1 to be absolutely sure. I don't have time to deploy a dev version of numpy to test.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/1208#issuecomment-273561115, or mute the thread https://github.com/notifications/unsubscribe-auth/ABKS1qvrlv_WzrX9rpoAD5zPY58KNJEnks5rTlu2gaJpZM4LkFPn .

Were you able to verify that the xarray tests pass after the numpy fix?

Thanks for the report. My guess is that this is an issue with the bottleneck build -- the large float values (e.g., 1e+248) in the final tests suggest some sort of overflow and/or memory corruption. The values summed in these tests are random numbers between 0 and 1.

Unfortunately, I can't reduce this locally using the conda build of bottleneck 1.2.0 on OS X, and our build on Travis-CI (using Ubuntu and conda) is also succeeding. Do you have any more specific details that describe your test setup, other than using the pre-build bottleneck 1.2.0 package?

If my hypothesis is correct, this test on bottleneck might trigger a test failure in the ubuntu build process (but it passed in bottleneck's tests on TravisCI): https://github.com/kwgoodman/bottleneck/compare/master...shoyer:possible-reduce-bug?expand=1#diff-a0a3ffc22e0a63118ba4a18e4ab845fc