home / github

Menu
  • Search all tables
  • GraphQL API

issue_comments

Table actions
  • GraphQL API for issue_comments

10 rows where author_association = "MEMBER", issue = 279161550 and user = 306380 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 1

  • mrocklin · 10 ✖

issue 1

  • dask compute on reduction failes with ValueError · 10 ✖

author_association 1

  • MEMBER · 10 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
349765925 https://github.com/pydata/xarray/issues/1759#issuecomment-349765925 https://api.github.com/repos/pydata/xarray/issues/1759 MDEyOklzc3VlQ29tbWVudDM0OTc2NTkyNQ== mrocklin 306380 2017-12-06T20:32:58Z 2017-12-06T20:32:58Z MEMBER

That seems sensible to me. It would also be a good way to ensure that XArray operations adhere to all of the dask.array checks.

from dask.array.utils import assert_eq
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  dask compute on reduction failes with ValueError 279161550
349630978 https://github.com/pydata/xarray/issues/1759#issuecomment-349630978 https://api.github.com/repos/pydata/xarray/issues/1759 MDEyOklzc3VlQ29tbWVudDM0OTYzMDk3OA== mrocklin 306380 2017-12-06T12:52:56Z 2017-12-06T12:52:56Z MEMBER

In the dask library itself we solve this by creating our own assert_eq functions that both invoke the single-threaded scheduler, and also do a variety of other sanity checks like ensuring that the expected and computed dtypes and shapes are the same, that the keynames in graphs are sensible, etc..

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  dask compute on reduction failes with ValueError 279161550
349452364 https://github.com/pydata/xarray/issues/1759#issuecomment-349452364 https://api.github.com/repos/pydata/xarray/issues/1759 MDEyOklzc3VlQ29tbWVudDM0OTQ1MjM2NA== mrocklin 306380 2017-12-05T21:47:18Z 2017-12-05T21:47:18Z MEMBER

I can remove all the mock related code from test_dask.py entirely and test_dataarray_with_dask_coords still passes.

It was just a guess. Something wacky is certainly happening though. I recommend copying my code from my last comment and running pytest on it either in the root directory or in xarray/tests. I found that the outcome differed depending on location.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  dask compute on reduction failes with ValueError 279161550
349151434 https://github.com/pydata/xarray/issues/1759#issuecomment-349151434 https://api.github.com/repos/pydata/xarray/issues/1759 MDEyOklzc3VlQ29tbWVudDM0OTE1MTQzNA== mrocklin 306380 2017-12-05T00:14:13Z 2017-12-05T00:14:13Z MEMBER

I still don't understand what actually went wrong here. It looks like we have test coverage for calling compute on an xarray.DataArray (see test_dataarray_with_dask_coords), but even though the exact example from that test fails at the repl the test passes when called with pytest:

I experienced some odd behavior when testing this within the XArray test suite

This file would pass when within xarray/tests/ but would fail when within the root directory

```python import numpy as np import xarray as xr import dask

def test_dask_reduction(): data = xr.DataArray(np.random.random(size=(10, 2)), dims=['samples', 'features']).chunk((5, 2)) result = dask.compute(data.mean(axis=0)) ```

I suspect some odd behavior around mock, but that's probably due to a general bias/lack of understanding of that module.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  dask compute on reduction failes with ValueError 279161550
349151169 https://github.com/pydata/xarray/issues/1759#issuecomment-349151169 https://api.github.com/repos/pydata/xarray/issues/1759 MDEyOklzc3VlQ29tbWVudDM0OTE1MTE2OQ== mrocklin 306380 2017-12-05T00:12:38Z 2017-12-05T00:12:38Z MEMBER

See https://github.com/pydata/xarray/pull/1760 for a potential fix

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  dask compute on reduction failes with ValueError 279161550
349149217 https://github.com/pydata/xarray/issues/1759#issuecomment-349149217 https://api.github.com/repos/pydata/xarray/issues/1759 MDEyOklzc3VlQ29tbWVudDM0OTE0OTIxNw== mrocklin 306380 2017-12-05T00:01:20Z 2017-12-05T00:01:20Z MEMBER

Also worth pointing out that this is likely the kind of bug that would have been caught with static typing

On Mon, Dec 4, 2017 at 6:55 PM, Stephan Hoyer notifications@github.com wrote:

Any objection to having the .compute methods point to dask.compute if the dask.version is appropriate?

Yes, this seems like a small win.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/1759#issuecomment-349148049, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszNQKcKZIvhT1rkNGONxU3rIH6gwOks5s9IZygaJpZM4Q1Xng .

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  dask compute on reduction failes with ValueError 279161550
349145345 https://github.com/pydata/xarray/issues/1759#issuecomment-349145345 https://api.github.com/repos/pydata/xarray/issues/1759 MDEyOklzc3VlQ29tbWVudDM0OTE0NTM0NQ== mrocklin 306380 2017-12-04T23:43:05Z 2017-12-04T23:43:05Z MEMBER

Any objection to having the .compute methods point to dask.compute if the dask.__version__ is appropriate?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  dask compute on reduction failes with ValueError 279161550
349144660 https://github.com/pydata/xarray/issues/1759#issuecomment-349144660 https://api.github.com/repos/pydata/xarray/issues/1759 MDEyOklzc3VlQ29tbWVudDM0OTE0NDY2MA== mrocklin 306380 2017-12-04T23:40:18Z 2017-12-04T23:40:18Z MEMBER

Here is the problem. It was just a silly typo.

```diff diff --git a/xarray/core/dataarray.py b/xarray/core/dataarray.py index 0516b47..263860d 100644 --- a/xarray/core/dataarray.py +++ b/xarray/core/dataarray.py @@ -594,7 +594,7 @@ class DataArray(AbstractArray, BaseDataObject):

 @property
 def __dask_scheduler__(self):
  • return self._to_temp_dataset().dask_optimize
  • return self._to_temp_dataset().dask_scheduler

    def dask_postcompute(self): func, args = self._to_temp_dataset().dask_postcompute() ```

@shoyer any suggestions on comprehensive ways to test things?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  dask compute on reduction failes with ValueError 279161550
349141548 https://github.com/pydata/xarray/issues/1759#issuecomment-349141548 https://api.github.com/repos/pydata/xarray/issues/1759 MDEyOklzc3VlQ29tbWVudDM0OTE0MTU0OA== mrocklin 306380 2017-12-04T23:25:43Z 2017-12-04T23:25:43Z MEMBER

While testing this I oddly learn that the following line makes this pass

python from xarray.tests import mock

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  dask compute on reduction failes with ValueError 279161550
349139597 https://github.com/pydata/xarray/issues/1759#issuecomment-349139597 https://api.github.com/repos/pydata/xarray/issues/1759 MDEyOklzc3VlQ29tbWVudDM0OTEzOTU5Nw== mrocklin 306380 2017-12-04T23:17:47Z 2017-12-04T23:17:47Z MEMBER

I'll take a look

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  dask compute on reduction failes with ValueError 279161550

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);
Powered by Datasette · Queries took 2239.308ms · About: xarray-datasette