html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue https://github.com/pydata/xarray/issues/1208#issuecomment-275956940,https://api.github.com/repos/pydata/xarray/issues/1208,275956940,MDEyOklzc3VlQ29tbWVudDI3NTk1Njk0MA==,10050469,2017-01-29T23:56:04Z,2017-01-29T23:56:04Z,MEMBER,"I added a PR in order to allow a dev version of bottleneck to be used, too: https://github.com/pydata/xarray/issues/1235","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,200908727 https://github.com/pydata/xarray/issues/1208#issuecomment-275955436,https://api.github.com/repos/pydata/xarray/issues/1208,275955436,MDEyOklzc3VlQ29tbWVudDI3NTk1NTQzNg==,1217238,2017-01-29T23:31:29Z,2017-01-29T23:31:29Z,MEMBER,"@fmaussion thanks for puzzling this one out! @ghisvail thanks for the report!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,200908727 https://github.com/pydata/xarray/issues/1208#issuecomment-275955256,https://api.github.com/repos/pydata/xarray/issues/1208,275955256,MDEyOklzc3VlQ29tbWVudDI3NTk1NTI1Ng==,10050469,2017-01-29T23:28:18Z,2017-01-29T23:28:18Z,MEMBER,the tests now pass with bottleneck master.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,200908727 https://github.com/pydata/xarray/issues/1208#issuecomment-275910605,https://api.github.com/repos/pydata/xarray/issues/1208,275910605,MDEyOklzc3VlQ29tbWVudDI3NTkxMDYwNQ==,10050469,2017-01-29T12:26:03Z,2017-01-29T12:26:03Z,MEMBER,"@shoyer nevermind, I found the bug: https://github.com/kwgoodman/bottleneck/issues/161 Quite a tricky one indeed ;-)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,200908727 https://github.com/pydata/xarray/issues/1208#issuecomment-275909069,https://api.github.com/repos/pydata/xarray/issues/1208,275909069,MDEyOklzc3VlQ29tbWVudDI3NTkwOTA2OQ==,10050469,2017-01-29T11:53:27Z,2017-01-29T11:53:27Z,MEMBER,"@shoyer there is something very weird going on. See the following example: ```python import numpy as np import bottleneck as bn import xarray as xr da = xr.DataArray(np.ones((10, 20)).astype(np.int), dims=['x', 'y'], coords={'abc':('y', np.array(['a'] * 9 + ['c'] + ['b'] * 10))}) np.testing.assert_allclose(np.sum(da[:, 9:10]), bn.nansum(da[:, 9:10])) # this will allways work np.testing.assert_allclose(da.groupby('abc').reduce(np.sum), da.groupby('abc').sum()) # this won't ``` This will work with bottleneck installed. Now change the ``astype(np.int)`` to ``astype(np.float)`` and the first assertion would still pass, but the second won't! It will fail with: ``` --------------------------------------------------------------------------- AssertionError Traceback (most recent call last) in () 2 coords={'abc':('y', np.array(['a'] * 9 + ['c'] + ['b'] * 10))}) 3 np.testing.assert_allclose(np.sum(da[:, 9:10]), bn.nansum(da[:, 9:10])) ----> 4 np.testing.assert_allclose(da.groupby('abc').reduce(np.sum), da.groupby('abc').sum()) /home/mowglie/.pyvirtualenvs/py3/lib/python3.4/site-packages/numpy/testing/utils.py in assert_allclose(actual, desired, rtol, atol, equal_nan, err_msg, verbose) 1409 header = 'Not equal to tolerance rtol=%g, atol=%g' % (rtol, atol) 1410 assert_array_compare(compare, actual, desired, err_msg=str(err_msg), -> 1411 verbose=verbose, header=header, equal_nan=equal_nan) 1412 1413 /home/mowglie/.pyvirtualenvs/py3/lib/python3.4/site-packages/numpy/testing/utils.py in assert_array_compare(comparison, x, y, err_msg, verbose, header, precision, equal_nan) 794 names=('x', 'y'), precision=precision) 795 if not cond: --> 796 raise AssertionError(msg) 797 except ValueError: 798 import traceback AssertionError: Not equal to tolerance rtol=1e-07, atol=0 (mismatch 33.33333333333333%) x: array([ 90., 100., 10.]) y: array([ 90., 100., 1.]) ``` So the new operation is applied only to the first element of the grouped selection, and I didn't manage to us bottleneck only to replicate this (since the first assertion always passes). Before I dig into this I'd like to have your opinion on this: any idea on what could go wrong here? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,200908727 https://github.com/pydata/xarray/issues/1208#issuecomment-275448697,https://api.github.com/repos/pydata/xarray/issues/1208,275448697,MDEyOklzc3VlQ29tbWVudDI3NTQ0ODY5Nw==,1217238,2017-01-26T17:14:26Z,2017-01-26T17:14:26Z,MEMBER,"@ghisvail Thanks for your diligence on this. @fmaussion If you can turn one of these into a test case for bottleneck to report upstream that would be super helpful. I would probably start with `test_groupby_sum`. It's likely that this only occurs for arrays with a particular `strides` (memory layout) and `shape`, which is where my [blind guess](https://github.com/kwgoodman/bottleneck/issues/160#issuecomment-272991823) that I suggested on the bottleneck tracker was inspired by.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,200908727 https://github.com/pydata/xarray/issues/1208#issuecomment-275396760,https://api.github.com/repos/pydata/xarray/issues/1208,275396760,MDEyOklzc3VlQ29tbWVudDI3NTM5Njc2MA==,10050469,2017-01-26T14:10:48Z,2017-01-26T14:10:48Z,MEMBER,I can confirm that - I have the same problems on my pip virtualenv on linux mint (I tried to reproduce it on [travis](https://github.com/pydata/xarray/pull/1222#issuecomment-274188191) without success),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,200908727 https://github.com/pydata/xarray/issues/1208#issuecomment-273569412,https://api.github.com/repos/pydata/xarray/issues/1208,273569412,MDEyOklzc3VlQ29tbWVudDI3MzU2OTQxMg==,1217238,2017-01-18T19:06:58Z,2017-01-18T19:06:58Z,MEMBER,"OK, thanks for looking into this! On Wed, Jan 18, 2017 at 10:36 AM, Ghislain Antony Vaillant < notifications@github.com> wrote: > We'd need to wait for numpy-1.12.1 to be absolutely sure. I don't have > time to deploy a dev version of numpy to test. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > , or mute > the thread > > . > ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,200908727 https://github.com/pydata/xarray/issues/1208#issuecomment-273560457,https://api.github.com/repos/pydata/xarray/issues/1208,273560457,MDEyOklzc3VlQ29tbWVudDI3MzU2MDQ1Nw==,1217238,2017-01-18T18:34:05Z,2017-01-18T18:34:05Z,MEMBER,Were you able to verify that the xarray tests pass after the numpy fix?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,200908727 https://github.com/pydata/xarray/issues/1208#issuecomment-272763117,https://api.github.com/repos/pydata/xarray/issues/1208,272763117,MDEyOklzc3VlQ29tbWVudDI3Mjc2MzExNw==,1217238,2017-01-16T02:58:06Z,2017-01-16T02:58:06Z,MEMBER,"Thanks for the report. My *guess* is that this is an issue with the bottleneck build -- the large float values (e.g., 1e+248) in the final tests suggest some sort of overflow and/or memory corruption. The values summed in these tests are random numbers between 0 and 1. Unfortunately, I can't reduce this locally using the conda build of bottleneck 1.2.0 on OS X, and our build on Travis-CI (using Ubuntu and conda) is also succeeding. Do you have any more specific details that describe your test setup, other than using the pre-build bottleneck 1.2.0 package? If my hypothesis is correct, this test on bottleneck might trigger a test failure in the ubuntu build process (but it passed in bottleneck's tests on TravisCI): https://github.com/kwgoodman/bottleneck/compare/master...shoyer:possible-reduce-bug?expand=1#diff-a0a3ffc22e0a63118ba4a18e4ab845fc ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,200908727