issue_comments

Table actions

9 rows where issue = 225734529 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 4

author_association 2

issue 1

  • autoclose with distributed doesn't seem to work · 9 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
453858625 https://github.com/pydata/xarray/issues/1394#issuecomment-453858625 https://api.github.com/repos/pydata/xarray/issues/1394 MDEyOklzc3VlQ29tbWVudDQ1Mzg1ODYyNQ== jhamman 2443309 2019-01-13T19:35:10Z 2019-01-13T19:35:10Z MEMBER

Closing this old issue. I'm assuming this behavior no longer exists following the backend refactors in 2018. @rabernat (or others) please reopen if you feel there is more to do here. 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   autoclose with distributed doesn't seem to work 225734529
298737596 https://github.com/pydata/xarray/issues/1394#issuecomment-298737596 https://api.github.com/repos/pydata/xarray/issues/1394 MDEyOklzc3VlQ29tbWVudDI5ODczNzU5Ng== rabernat 1197350 2017-05-02T19:33:38Z 2017-05-02T19:33:51Z MEMBER

I think that there is an underlying problem with the way that open_mfdataset is building the dask graph for this dataset (see #1396). Operations seem overly eager to read all the data and load it into memory. So it might not be a problem with autoclose after all.

I do notice that autoclose does work in certain cases. For example, after I open the dataset, it doesn't leave the files open. That's good.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   autoclose with distributed doesn't seem to work 225734529
298735693 https://github.com/pydata/xarray/issues/1394#issuecomment-298735693 https://api.github.com/repos/pydata/xarray/issues/1394 MDEyOklzc3VlQ29tbWVudDI5ODczNTY5Mw== rabernat 1197350 2017-05-02T19:25:59Z 2017-05-02T19:25:59Z MEMBER

netCDF4. decode_cf doesn't seem to affect anything important.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   autoclose with distributed doesn't seem to work 225734529
298735223 https://github.com/pydata/xarray/issues/1394#issuecomment-298735223 https://api.github.com/repos/pydata/xarray/issues/1394 MDEyOklzc3VlQ29tbWVudDI5ODczNTIyMw== pwolfram 4295853 2017-05-02T19:24:07Z 2017-05-02T19:24:07Z CONTRIBUTOR

Note, we don't use decode_cf=False. Does it crash without making this specification, e.g., using the default?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   autoclose with distributed doesn't seem to work 225734529
298735070 https://github.com/pydata/xarray/issues/1394#issuecomment-298735070 https://api.github.com/repos/pydata/xarray/issues/1394 MDEyOklzc3VlQ29tbWVudDI5ODczNTA3MA== pwolfram 4295853 2017-05-02T19:23:30Z 2017-05-02T19:23:30Z CONTRIBUTOR

@rabernat, I would say that this is a bug. Is this with the scipy backend or netCDF4? Presumably if you have this problem we could run into too. For the record, we are using netCDF4.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   autoclose with distributed doesn't seem to work 225734529
298692920 https://github.com/pydata/xarray/issues/1394#issuecomment-298692920 https://api.github.com/repos/pydata/xarray/issues/1394 MDEyOklzc3VlQ29tbWVudDI5ODY5MjkyMA== rabernat 1197350 2017-05-02T16:48:32Z 2017-05-02T16:48:32Z MEMBER

The idea of a fuller test suite is a good idea.

One problem is that many of these applications involve really big datasets, so it is hard to share examples.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   autoclose with distributed doesn't seem to work 225734529
298688491 https://github.com/pydata/xarray/issues/1394#issuecomment-298688491 https://api.github.com/repos/pydata/xarray/issues/1394 MDEyOklzc3VlQ29tbWVudDI5ODY4ODQ5MQ== shoyer 1217238 2017-05-02T16:32:07Z 2017-05-02T16:32:07Z MEMBER

0.9.3

OK, so that shouldn't be a problem. Hmm.

My only suggestion is that we should think about trying to write a fuller test suite for the auto-close functionality, using a mock or fake of some sort that we can interrogate to verify it works properly. One simple thing would be to refactor the autoclose functionality into a single separate adaptor datastore (which wraps an underlying datastore) that we can more easily test, rather than putting it onto each of the underlying datastore classes. I'm not sure why I didn't think of that when @pwolfram was writing this before.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   autoclose with distributed doesn't seem to work 225734529
298683384 https://github.com/pydata/xarray/issues/1394#issuecomment-298683384 https://api.github.com/repos/pydata/xarray/issues/1394 MDEyOklzc3VlQ29tbWVudDI5ODY4MzM4NA== rabernat 1197350 2017-05-02T16:13:56Z 2017-05-02T16:13:56Z MEMBER

0.9.3

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   autoclose with distributed doesn't seem to work 225734529
298681206 https://github.com/pydata/xarray/issues/1394#issuecomment-298681206 https://api.github.com/repos/pydata/xarray/issues/1394 MDEyOklzc3VlQ29tbWVudDI5ODY4MTIwNg== shoyer 1217238 2017-05-02T16:06:18Z 2017-05-02T16:06:18Z MEMBER

Just to make sure, which version of xarray are you using?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   autoclose with distributed doesn't seem to work 225734529

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);