issue_comments

Table actions

10 rows where issue = 286542795 and user = 2443309 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date)

user 1

  • jhamman · 10 ✖

issue 1

  • WIP: Compute==False for to_zarr and to_netcdf · 10 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
389553054 https://github.com/pydata/xarray/pull/1811#issuecomment-389553054 https://api.github.com/repos/pydata/xarray/issues/1811 MDEyOklzc3VlQ29tbWVudDM4OTU1MzA1NA== jhamman 2443309 2018-05-16T15:06:51Z 2018-05-16T15:06:51Z MEMBER

Thanks all for the input/reviews on this PR. 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   WIP: Compute==False for to_zarr and to_netcdf 286542795
389294623 https://github.com/pydata/xarray/pull/1811#issuecomment-389294623 https://api.github.com/repos/pydata/xarray/issues/1811 MDEyOklzc3VlQ29tbWVudDM4OTI5NDYyMw== jhamman 2443309 2018-05-15T20:00:58Z 2018-05-15T20:00:58Z MEMBER

I'm little surprised it doesn't just work with scipy and h5netcdf -- have you tried them again recently?

Yes, just tried again. I'm open to ideas but would also like to move this issue along first, if possible.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   WIP: Compute==False for to_zarr and to_netcdf 286542795
388984522 https://github.com/pydata/xarray/pull/1811#issuecomment-388984522 https://api.github.com/repos/pydata/xarray/issues/1811 MDEyOklzc3VlQ29tbWVudDM4ODk4NDUyMg== jhamman 2443309 2018-05-14T22:34:48Z 2018-05-14T22:34:48Z MEMBER

@shoyer / @mrocklin - I think this is ready for another review. Since I asked for reviews last, I have:

  • reworked the task graph to write one file / close one file rather than write all / close all.
  • moved the tests to the TestDask class which seemed more appropriate. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   WIP: Compute==False for to_zarr and to_netcdf 286542795
388649669 https://github.com/pydata/xarray/pull/1811#issuecomment-388649669 https://api.github.com/repos/pydata/xarray/issues/1811 MDEyOklzc3VlQ29tbWVudDM4ODY0OTY2OQ== jhamman 2443309 2018-05-13T19:20:25Z 2018-05-13T19:20:25Z MEMBER

Actually, scratch that - I just found https://github.com/pydata/xarray/pull/1811/files#r183091638 which explains what is goin on. Sorry for the noise

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   WIP: Compute==False for to_zarr and to_netcdf 286542795
388649561 https://github.com/pydata/xarray/pull/1811#issuecomment-388649561 https://api.github.com/repos/pydata/xarray/issues/1811 MDEyOklzc3VlQ29tbWVudDM4ODY0OTU2MQ== jhamman 2443309 2018-05-13T19:18:37Z 2018-05-13T19:18:47Z MEMBER

@shoyer - I'm getting a test failure in the h5netcdf backend that seems unrelated. Do you know if something has changed in the string handling of h5netcdf recently?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   WIP: Compute==False for to_zarr and to_netcdf 286542795
383077508 https://github.com/pydata/xarray/pull/1811#issuecomment-383077508 https://api.github.com/repos/pydata/xarray/issues/1811 MDEyOklzc3VlQ29tbWVudDM4MzA3NzUwOA== jhamman 2443309 2018-04-20T12:16:43Z 2018-04-20T12:16:43Z MEMBER

The test failures in the latest build appear to be unrelated to this PR.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   WIP: Compute==False for to_zarr and to_netcdf 286542795
382488891 https://github.com/pydata/xarray/pull/1811#issuecomment-382488891 https://api.github.com/repos/pydata/xarray/issues/1811 MDEyOklzc3VlQ29tbWVudDM4MjQ4ODg5MQ== jhamman 2443309 2018-04-18T18:43:23Z 2018-04-18T18:43:23Z MEMBER

I see you were already using the LocalCluster (Client). Disregard my comment on switching clusters. I seem to be getting my Github issues mixed up.

It may be good to take this offline to one of the pangeo/zarr/dask/xarray issues (there are a few). 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   WIP: Compute==False for to_zarr and to_netcdf 286542795
382450276 https://github.com/pydata/xarray/pull/1811#issuecomment-382450276 https://api.github.com/repos/pydata/xarray/issues/1811 MDEyOklzc3VlQ29tbWVudDM4MjQ1MDI3Ng== jhamman 2443309 2018-04-18T16:37:12Z 2018-04-18T16:37:12Z MEMBER

@rsignell-usgs - This is going to require some debugging on your part but here are a few suggestions:

  • try smaller chunksize
  • use a LocalCluster before using the KubeCluster
  • turn up dask's logging level for the scheduler/workers

and after 20 seconds or so, the process dies with this error:

looking above, I don't see any error. Your goal in the next step should be to find the error. If workers are dying, they should report that in the worker logs. If your cluster is dying, that should be reported in the notebook. 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   WIP: Compute==False for to_zarr and to_netcdf 286542795
382169654 https://github.com/pydata/xarray/pull/1811#issuecomment-382169654 https://api.github.com/repos/pydata/xarray/issues/1811 MDEyOklzc3VlQ29tbWVudDM4MjE2OTY1NA== jhamman 2443309 2018-04-17T22:04:31Z 2018-04-17T22:04:31Z MEMBER

@rsignell-usgs - can you repeat this example with a single threaded scheduler. It will be slow but it should work (or return a more informative error). 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   WIP: Compute==False for to_zarr and to_netcdf 286542795
375797389 https://github.com/pydata/xarray/pull/1811#issuecomment-375797389 https://api.github.com/repos/pydata/xarray/issues/1811 MDEyOklzc3VlQ29tbWVudDM3NTc5NzM4OQ== jhamman 2443309 2018-03-23T21:02:11Z 2018-03-23T21:02:11Z MEMBER

@shoyer - I did my best to implement the finalize store concept you've described above. This all seems to work nicely except for h5netcdf. During the store operation, h5netcdf is raising the following error:

```Python self = <h5netcdf.core._LazyObjectLookup object at 0x122d8b748>, key = 'foo'

def __getitem__(self, key):

  if self._objects[key] is not None:

E KeyError: 'foo' ```

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   WIP: Compute==False for to_zarr and to_netcdf 286542795

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);