issue_comments

Table actions

2 rows where author_association = "NONE" and issue = 376370028 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 2

issue 1

  • to_dataframe() excessive memory usage · 2 ✖

author_association 1

  • NONE · 2 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1114305976 https://github.com/pydata/xarray/issues/2534#issuecomment-1114305976 https://api.github.com/repos/pydata/xarray/issues/2534 IC_kwDOAMm_X85CavW4 stale[bot] 26384082 2022-05-01T18:37:45Z 2022-05-01T18:37:45Z NONE

In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity

If this issue remains relevant, please comment here or remove the stale label; otherwise it will be marked as closed automatically

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   to_dataframe() excessive memory usage 376370028
613068333 https://github.com/pydata/xarray/issues/2534#issuecomment-613068333 https://api.github.com/repos/pydata/xarray/issues/2534 MDEyOklzc3VlQ29tbWVudDYxMzA2ODMzMw== sgdecker 8419421 2020-04-13T19:57:08Z 2020-04-13T19:57:08Z NONE

Here is another example: import xarray as xr ncdata = xr.open_dataset('https://thredds.ucar.edu/thredds/dodsC/nws/metar/ncdecoded/files/Surface_METAR_20200411_0000.nc') df = ncdata.to_dataframe()

Output is: Traceback (most recent call last): File "bug.py", line 4, in <module> df = ncdata.to_dataframe() File "/home/decker/miniconda3/envs/met212/lib/python3.8/site-packages/xarray/core/dataset.py", line 4399, in to_dataframe return self._to_dataframe(self.dims) File "/home/decker/miniconda3/envs/met212/lib/python3.8/site-packages/xarray/core/dataset.py", line 4385, in _to_dataframe data = [ File "/home/decker/miniconda3/envs/met212/lib/python3.8/site-packages/xarray/core/dataset.py", line 4386, in <listcomp> self._variables[k].set_dims(ordered_dims).values.reshape(-1) MemoryError: Unable to allocate array with shape (117819, 5021) and data type |S64

If I'm doing the math right, xarray is trying to allocate roughly 35 GB even though this NetCDF file is only on the order of 50 MB in size.

Output of xr.show_versions()

Details ``` INSTALLED VERSIONS ------------------ commit: None python: 3.8.1 | packaged by conda-forge | (default, Jan 5 2020, 20:58:18) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 5.4.31-1-MANJARO machine: x86_64 processor: byteorder: little LC_ALL: None LANG: en_US.utf8 LOCALE: en_US.UTF-8 libhdf5: 1.10.5 libnetcdf: 4.7.3 xarray: 0.14.1 pandas: 0.25.3 numpy: 1.17.5 scipy: 1.4.1 netCDF4: 1.5.3 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.0.4.2 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: None distributed: None matplotlib: 3.1.2 cartopy: 0.17.0 seaborn: None numbagg: None setuptools: 45.1.0.post20200119 pip: 19.3.1 conda: None pytest: 5.3.4 IPython: 7.11.1 sphinx: None ``` 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   to_dataframe() excessive memory usage 376370028

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);