issue_comments

Table actions

5 rows where author_association = "MEMBER" and issue = 326533369 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 3

issue 1

  • Memory leak while looping through a Dataset · 5 ✖

author_association 1

  • MEMBER · 5 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1035611864 https://github.com/pydata/xarray/issues/2186#issuecomment-1035611864 https://api.github.com/repos/pydata/xarray/issues/2186 IC_kwDOAMm_X849ui7Y shoyer 1217238 2022-02-10T22:49:40Z 2022-02-10T22:50:01Z MEMBER

For what it's wroth, the recommended way to do this is to explicitly close the Dataset with ds.close() rather than using del ds.

Or with a context manager, e.g., python for num in range(100): with xr.open_dataset('data.{}.nc'.format(num)) as ds: # do some stuff, but NOT assigning any data in ds to new variables ...

 
{
    "total_count": 2,
    "+1": 2,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Memory leak while looping through a Dataset 326533369
454162269 https://github.com/pydata/xarray/issues/2186#issuecomment-454162269 https://api.github.com/repos/pydata/xarray/issues/2186 MDEyOklzc3VlQ29tbWVudDQ1NDE2MjI2OQ== max-sixty 5635139 2019-01-14T21:09:36Z 2019-01-14T21:09:36Z MEMBER

In an effort to reduce the issue backlog, I'll close this, but please reopen if you disagree

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Memory leak while looping through a Dataset 326533369
393996561 https://github.com/pydata/xarray/issues/2186#issuecomment-393996561 https://api.github.com/repos/pydata/xarray/issues/2186 MDEyOklzc3VlQ29tbWVudDM5Mzk5NjU2MQ== shoyer 1217238 2018-06-01T20:13:18Z 2018-06-01T20:13:18Z MEMBER

This might be the same issue as https://github.com/dask/dask/issues/3530

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Memory leak while looping through a Dataset 326533369
392234293 https://github.com/pydata/xarray/issues/2186#issuecomment-392234293 https://api.github.com/repos/pydata/xarray/issues/2186 MDEyOklzc3VlQ29tbWVudDM5MjIzNDI5Mw== shoyer 1217238 2018-05-26T03:58:14Z 2018-05-26T03:58:14Z MEMBER

I might try experimenting with setting autoclose=True in open_mfdataset(). It's a bit of a short in the dark, but that might help here.

Memory growth with xr.open_dataset('data.nc', chunks=None) is expected, because by default we set cache=True when not using dask. This means that variables get cached in memory as NumPy arrays.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Memory leak while looping through a Dataset 326533369
392108417 https://github.com/pydata/xarray/issues/2186#issuecomment-392108417 https://api.github.com/repos/pydata/xarray/issues/2186 MDEyOklzc3VlQ29tbWVudDM5MjEwODQxNw== rabernat 1197350 2018-05-25T16:17:15Z 2018-05-25T16:17:15Z MEMBER

The memory management here is handled by python, not xarray. Python decides when to perform garbage collection. I know that doesn't help solve your problem...

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Memory leak while looping through a Dataset 326533369

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);