issue_comments

Table actions

3 rows where issue = 1465047346 and user = 83403825 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

  • adanb13 · 3 ✖

issue 1

  • (Issue #7324) added functions that return data values in memory efficient manner · 3 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1419917480 https://github.com/pydata/xarray/pull/7323#issuecomment-1419917480 https://api.github.com/repos/pydata/xarray/issues/7323 IC_kwDOAMm_X85Uojio adanb13 83403825 2023-02-06T23:10:39Z 2023-02-06T23:10:39Z NONE

@jhamman yes, I think it's alright to close. The issue seems to arise from the use of .tolist(), having to convert every value to a python float so that it may be retuned in a dictionary as is acceptable for JSON, causing the memory spike.

will try @Illviljan suggestion (thanks!) , to_json calls .compute but not sure if it calls .tolist(), in case it avoids it, that should result in greater memory efficiency. 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   (Issue #7324) added functions that return data values in memory efficient manner 1465047346
1328142597 https://github.com/pydata/xarray/pull/7323#issuecomment-1328142597 https://api.github.com/repos/pydata/xarray/issues/7323 IC_kwDOAMm_X85PKdkF adanb13 83403825 2022-11-27T00:48:34Z 2022-11-27T02:00:10Z NONE

I'm not sure if this breaks the data model of xarray leaving inconsistent sizes?

Also this seems like a very corner usecase, I don't think it is intended to write DataArrays in ASCII.

But I let some more senior devs of xarray be the judge here :)

Made these for work (big data, government). Is useful when trying to provide data values back to end user after all data manipulation has been done. (Aka the initial Xarray.DataArray is not longer needed)

best native solution that exists (from what I see) is .to_dict() which is memory inefficient. (Had memory errors at work when trying, hence all the tests provided in the gist). Basically a user can call to_dict with data = false and then add the data values using the above 2 functions to the resulting dictionary in a more memory efficient way.

Use cases would be in any web service that would like to provide the final data values back to a user in JSON.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   (Issue #7324) added functions that return data values in memory efficient manner 1465047346
1327982136 https://github.com/pydata/xarray/pull/7323#issuecomment-1327982136 https://api.github.com/repos/pydata/xarray/issues/7323 IC_kwDOAMm_X85PJ2Y4 adanb13 83403825 2022-11-26T05:09:04Z 2022-11-26T05:09:04Z NONE

ran python -m pip uninstall urllib3-secure-extra for Doctests fail as suggested. Get the following message: WARNING: Skipping urllib3-secure-extra as it is not installed.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   (Issue #7324) added functions that return data values in memory efficient manner 1465047346

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);