issues

Table actions

2 rows where user = 83403825 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date), closed_at (date)

type 2

state 1

repo 1

id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
1465230135 I_kwDOAMm_X85XVaM3 7324 Memory issues arising when trying to make dataArray values JSON serializable adanb13 83403825 closed 0     2 2022-11-26T18:54:06Z 2023-09-12T15:18:14Z 2023-09-12T15:18:13Z NONE      

What is your issue?

Need to return the data values in a JSON serializable format, calling .tolist and .to_dict consume so much memory that it produces a Memory Error

When trying to get the data values via the 2 methods above from a dataset the size of 831 MiBs the system memory being used jumped to 43 G before crashing (on a system of 64 G memory capacity). Producing a Memory Error

Need more memory efficient way of returning final data values.

Addressed in the following PR: https://github.com/pydata/xarray/pull/7323

 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/7324/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed xarray 13221727 issue
1465047346 PR_kwDOAMm_X85Du-Co 7323 (Issue #7324) added functions that return data values in memory efficient manner adanb13 83403825 closed 0     10 2022-11-26T04:54:50Z 2023-02-07T18:25:18Z 2023-02-07T18:25:17Z NONE   0 pydata/xarray/pulls/7323

added functions: - data_to_list_save_memory - _genlist

data_to_list_save_memory returns a list of data values from a given Xarray.DataArray. Creates a list of data values as data values from the original Xarray.DataArray is removed (saving memory)

_genlist provides a generator (Most memory efficient)

Useful when returning data values in json format of large datasets after the dataset no longer in use to avoid memory issues. The function is more efficient than to_dict() regarding memory usage when calling for just the data values.

 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/7323/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
     xarray 13221727 pull

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [active_lock_reason] TEXT,
   [draft] INTEGER,
   [pull_request] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [state_reason] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
);
CREATE INDEX [idx_issues_repo]
    ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
    ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
    ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
    ON [issues] ([user]);