issue_comments

Table actions

9 rows where author_association = "MEMBER" and issue = 218459353 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

issue 1

  • bottleneck : Wrong mean for float32 array · 9 ✖

author_association 1

  • MEMBER · 9 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1119787557 https://github.com/pydata/xarray/issues/1346#issuecomment-1119787557 https://api.github.com/repos/pydata/xarray/issues/1346 IC_kwDOAMm_X85Cvpol dcherian 2448579 2022-05-06T16:22:32Z 2022-05-06T16:22:32Z MEMBER

On second thought we should add this to a FAQ page.

 
{
    "total_count": 2,
    "+1": 2,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   bottleneck : Wrong mean for float32 array 218459353
1119786892 https://github.com/pydata/xarray/issues/1346#issuecomment-1119786892 https://api.github.com/repos/pydata/xarray/issues/1346 IC_kwDOAMm_X85CvpeM dcherian 2448579 2022-05-06T16:21:42Z 2022-05-06T16:21:42Z MEMBER

Yes that sounds right. Thanks!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   bottleneck : Wrong mean for float32 array 218459353
1119770101 https://github.com/pydata/xarray/issues/1346#issuecomment-1119770101 https://api.github.com/repos/pydata/xarray/issues/1346 IC_kwDOAMm_X85CvlX1 andersy005 13301940 2022-05-06T16:01:44Z 2022-05-06T16:01:44Z MEMBER 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   bottleneck : Wrong mean for float32 array 218459353
464115604 https://github.com/pydata/xarray/issues/1346#issuecomment-464115604 https://api.github.com/repos/pydata/xarray/issues/1346 MDEyOklzc3VlQ29tbWVudDQ2NDExNTYwNA== shoyer 1217238 2019-02-15T16:39:08Z 2019-02-15T16:39:08Z MEMBER

The difference is that Bottleneck does the sum in the naive way, whereas NumPy uses the more numerically stable pairwise summation.

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   bottleneck : Wrong mean for float32 array 218459353
456173428 https://github.com/pydata/xarray/issues/1346#issuecomment-456173428 https://api.github.com/repos/pydata/xarray/issues/1346 MDEyOklzc3VlQ29tbWVudDQ1NjE3MzQyOA== shoyer 1217238 2019-01-21T19:09:43Z 2019-01-21T19:09:43Z MEMBER

Would it be worth adding a warning (until the right solution is found) if someone is doing .mean() on a DataArray which is float32?

I would rather pick option (1) above, that is, "Stop using bottleneck on float32 arrays"

 
{
    "total_count": 3,
    "+1": 3,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   bottleneck : Wrong mean for float32 array 218459353
290851733 https://github.com/pydata/xarray/issues/1346#issuecomment-290851733 https://api.github.com/repos/pydata/xarray/issues/1346 MDEyOklzc3VlQ29tbWVudDI5MDg1MTczMw== shoyer 1217238 2017-03-31T22:55:18Z 2017-03-31T22:55:18Z MEMBER

@matteodefelice you didn't decide on float32, but your data is stored that way. It's really hard to make choices about numerical precision for computations automatically: if we converted automatically to float64, somebody else would be complaining about unexpected memory usage :).

Looking at our options, we could:

  1. Stop using bottleneck on float32 arrays, or provide a flag or option to disable using bottleneck. This is not ideal, because bottleneck is much faster.
  2. Automatically convert float32 arrays to float64 before doing aggregations. This is not ideal, because it could significant increase memory requirements.
  3. Add a dtype option for aggregations (like NumPy) and consider defaulting to dype=np.float64 when doing aggregations on float32 arrays. I would generally be happy with this, but bottleneck currently doesn't provide the option currently.
  4. Write a higher precision algorithm for bottleneck's mean.
 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   bottleneck : Wrong mean for float32 array 218459353
290760342 https://github.com/pydata/xarray/issues/1346#issuecomment-290760342 https://api.github.com/repos/pydata/xarray/issues/1346 MDEyOklzc3VlQ29tbWVudDI5MDc2MDM0Mg== shoyer 1217238 2017-03-31T16:24:04Z 2017-03-31T16:24:04Z MEMBER

Yes, this is probably related to the fact that .mean() in xarray uses bottleneck if available, and bottleneck has a slightly different mean implementation, quite possibly with a less numerically stable algorithm.

The fact that the dtype is float32 is a sign that this is probably a numerical precision issue. Try casting with .astype(np.float64) and see if the problem goes away.

If you really cared about performance using float32, the other thing to do to improve conditioning is to subtract and add a number close to the mean, e.g., (ds.var167 - 270).mean() + 270.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   bottleneck : Wrong mean for float32 array 218459353
290754443 https://github.com/pydata/xarray/issues/1346#issuecomment-290754443 https://api.github.com/repos/pydata/xarray/issues/1346 MDEyOklzc3VlQ29tbWVudDI5MDc1NDQ0Mw== fmaussion 10050469 2017-03-31T16:02:53Z 2017-03-31T16:02:53Z MEMBER

Does it make a difference if you load the data first? (ds.var167.load().mean()) Or use python 3?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   bottleneck : Wrong mean for float32 array 218459353
290691941 https://github.com/pydata/xarray/issues/1346#issuecomment-290691941 https://api.github.com/repos/pydata/xarray/issues/1346 MDEyOklzc3VlQ29tbWVudDI5MDY5MTk0MQ== fmaussion 10050469 2017-03-31T11:50:05Z 2017-03-31T11:50:05Z MEMBER

I can't reproduce this:

```python In [6]: ds = xr.open_dataset('./Downloads/ERAIN-t2m-1983-2012.seasmean.nc')

In [7]: ds.var167.mean() Out[7]: <xarray.DataArray 'var167' ()> array(278.6246643066406, dtype=float32)

In [8]: ds.var167.data.mean() Out[8]: 278.62466 ```

which version of xarray, dask, python are you using?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   bottleneck : Wrong mean for float32 array 218459353

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);