issue_comments

Table actions

5 rows where author_association = "MEMBER" and issue = 243927150 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 4

issue 1

  • Excessive memory usage when printing multi-file Dataset · 5 ✖

author_association 1

  • MEMBER · 5 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
331539708 https://github.com/pydata/xarray/issues/1481#issuecomment-331539708 https://api.github.com/repos/pydata/xarray/issues/1481 MDEyOklzc3VlQ29tbWVudDMzMTUzOTcwOA== jhamman 2443309 2017-09-22T19:30:00Z 2017-09-22T19:30:00Z MEMBER

@hadfieldnz - I think this was just fixed in #1532. Keep an eye out for the 0.10 release. Feel free to reopen if you feel there's more to do here.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Excessive memory usage when printing multi-file Dataset 243927150
317050127 https://github.com/pydata/xarray/issues/1481#issuecomment-317050127 https://api.github.com/repos/pydata/xarray/issues/1481 MDEyOklzc3VlQ29tbWVudDMxNzA1MDEyNw== shoyer 1217238 2017-07-21T16:40:19Z 2017-07-21T16:40:19Z MEMBER

Our formatting logic pulls out the first few values of arrays to print them in the repr. It appears that this is failing spectacularly in this case, though I'm not sure why.

Can you share a quick preview of what a single one of your constituent netCDF files looks like?

More broadly: maybe we should disable automatically printing a preview of the contents of xarray.Dataset objects when they have lazily loaded data in the form of dask arrays. This is convenient for interactive use in many cases (when it can be done cheaply!) but fails in many edge cases.

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Excessive memory usage when printing multi-file Dataset 243927150
316989732 https://github.com/pydata/xarray/issues/1481#issuecomment-316989732 https://api.github.com/repos/pydata/xarray/issues/1481 MDEyOklzc3VlQ29tbWVudDMxNjk4OTczMg== rabernat 1197350 2017-07-21T12:37:11Z 2017-07-21T12:37:21Z MEMBER

Can you try calling open_mfdataset with the decode_cf=False option?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Excessive memory usage when printing multi-file Dataset 243927150
316937056 https://github.com/pydata/xarray/issues/1481#issuecomment-316937056 https://api.github.com/repos/pydata/xarray/issues/1481 MDEyOklzc3VlQ29tbWVudDMxNjkzNzA1Ng== fmaussion 10050469 2017-07-21T08:18:52Z 2017-07-21T08:18:52Z MEMBER

0.14.3 pre-dates the fix https://github.com/dask/dask/pull/2364 mentioned above: can you try to update dask?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Excessive memory usage when printing multi-file Dataset 243927150
316806744 https://github.com/pydata/xarray/issues/1481#issuecomment-316806744 https://api.github.com/repos/pydata/xarray/issues/1481 MDEyOklzc3VlQ29tbWVudDMxNjgwNjc0NA== rabernat 1197350 2017-07-20T19:32:18Z 2017-07-20T19:32:18Z MEMBER

Hi @hadfieldnz -- I believe this issue could be related to #1396, which was fixed in dask/dask#2364.

Could you let us know what versions of xarray and dask you are using?

python import xarray import dask print(xarray.__version__) print(dask.__version__)

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Excessive memory usage when printing multi-file Dataset 243927150

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);