issue_comments

Table actions

2 rows where issue = 1676561243 and user = 123355381 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

  • dabhicusp · 2 ✖

issue 1

  • Process getting killed due to high memory consumption of xarray's nbytes method · 2 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1519897098 https://github.com/pydata/xarray/issues/7772#issuecomment-1519897098 https://api.github.com/repos/pydata/xarray/issues/7772 IC_kwDOAMm_X85al8oK dabhicusp 123355381 2023-04-24T10:51:16Z 2023-04-24T10:51:16Z NONE

Thank you @dcherian . I cannot reproduced this on main.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Process getting killed due to high memory consumption of xarray's nbytes method 1676561243
1517649648 https://github.com/pydata/xarray/issues/7772#issuecomment-1517649648 https://api.github.com/repos/pydata/xarray/issues/7772 IC_kwDOAMm_X85adX7w dabhicusp 123355381 2023-04-21T10:57:28Z 2023-04-21T10:57:28Z NONE

The first point that you mentioned does not seem to be correct. Please see the below code (we took the sparse matrix ) and output: ``` import xarray as xa import numpy as np

def get_data(): lat_dim = 7210 lon_dim = 7440

lat = [0] * lat_dim
lon = [0] * lon_dim 
time = [0] * 5

nlats = lat_dim; nlons = lon_dim; ntimes = 5

var_1 = np.empty((ntimes,nlats,nlons))
var_2 = np.empty((ntimes,nlats,nlons))
var_3 = np.empty((ntimes,nlats,nlons))
var_4 = np.empty((ntimes,nlats,nlons))

data_arr = np.random.uniform(low=0,high=0,size=(ntimes,nlats,nlons))
data_arr[:,0,:] = 1
data_arr[:,:,1] = 1

var_1[:,:,:] = data_arr 
var_2[:,:,:] = data_arr 
var_3[:,:,:] = data_arr 
var_4[:,:,:] = data_arr

dataset = xa.Dataset( 
        data_vars = {
            'var_1': (('time','lat','lon'), var_1),
            'var_2': (('time','lat','lon'), var_2),
            'var_3': (('time','lat','lon'), var_3),
            'var_4': (('time','lat','lon'), var_4)},
        coords = {
            'lat': lat,
            'lon': lon,
            'time':time})

print(sum(v.size * v.dtype.itemsize for v in dataset.variables.values()))
print(dataset.nbytes)

if name == "main": get_data() ```

8582901240 8582901240 As we can observe here both nbytes and self.size * self.dtype.itemsize gives the same size.

And for the 2nd point can you share any solution for the nbytes for the netCDF or grib file as it takes too much memory and killed the process?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Process getting killed due to high memory consumption of xarray's nbytes method 1676561243

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);