issue_comments

Table actions

3 rows where issue = 184722754 and user = 6213168 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: updated_at (date)

user 1

  • crusaderky · 3 ✖

issue 1

  • shallow copies become deep copies when pickling · 3 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
277543644 https://github.com/pydata/xarray/issues/1058#issuecomment-277543644 https://api.github.com/repos/pydata/xarray/issues/1058 MDEyOklzc3VlQ29tbWVudDI3NzU0MzY0NA== crusaderky 6213168 2017-02-05T19:44:33Z 2017-02-05T19:44:33Z MEMBER

Actually, I very much still am facing the problem. The biggest issue is now when I need to invoke xarray.broadcast. In my use case, I'm broadcasting together

  • a scalar array with numpy backend, shape=(), chunks=None
  • a 1D array with dask backend, shape=(2**19,), chunks=(2**15,)

What broadcast does is transform the scalar array to a numpy array of 2**19 elements. This is actually a view on the original 0D array, so it's got negligible RAM requirements. But after pickling and unpickling, it's become a real 2**19 elements array. Add up a few hundreds of them, and I am facing GBs of wasted RAM.

A solution would be to change broadcast() to convert to dask before broadcasting, and then broadcast directly to the proper chunk size.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   shallow copies become deep copies when pickling 184722754
260773846 https://github.com/pydata/xarray/issues/1058#issuecomment-260773846 https://api.github.com/repos/pydata/xarray/issues/1058 MDEyOklzc3VlQ29tbWVudDI2MDc3Mzg0Ng== crusaderky 6213168 2016-11-15T21:26:52Z 2016-11-15T21:26:52Z MEMBER

Confirmed that #1017 fixes my specific issue, thanks! Leaving the ticket open as other people (particularly those that work on large arrays without dask) will still be affected.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   shallow copies become deep copies when pickling 184722754
255952251 https://github.com/pydata/xarray/issues/1058#issuecomment-255952251 https://api.github.com/repos/pydata/xarray/issues/1058 MDEyOklzc3VlQ29tbWVudDI1NTk1MjI1MQ== crusaderky 6213168 2016-10-25T06:54:02Z 2016-10-25T06:54:02Z MEMBER

@maximilianr, if you pickle 2 plain python objects A and B together, and one of the attributes of B is a reference to A, A does not get duplicated.

In this case there must be some specific getstate code to prevent this and/or something with the C implementation of the class 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   shallow copies become deep copies when pickling 184722754

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);