issues: 944996552
This data as json
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | active_lock_reason | draft | pull_request | body | reactions | performed_via_github_app | state_reason | repo | type |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
944996552 | MDU6SXNzdWU5NDQ5OTY1NTI= | 5604 | Extremely Large Memory usage for a very small variable | 49487505 | closed | 0 | 15 | 2021-07-15T04:52:35Z | 2023-09-12T15:31:13Z | 2023-09-12T15:31:12Z | NONE | What happened: Variable that takes up very little memory in actual data size uses over 1000x the memory What you expected to happen: It should take the order of the memory of the data size Minimal Complete Verifiable Example: ```python Put your MCVE code here
Anything else we need to know?: The variable u is able to be written to the disk with 22Gb of memory usage, which is the expected behaviour as the variable has abou 22 Gb of data stored in those files combined. In fact, seeing that the dimension of u is 35x ubar, the file size of u_file should be only about 22 to 23Gb. Environment: Output of <tt>xr.show_versions()</tt>INSTALLED VERSIONS ------------------ commit: None python: 3.8.8 (default, Apr 13 2021, 19:58:26) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 3.10.0-957.12.2.el7.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.12.0 libnetcdf: 4.7.4 xarray: 0.18.2 pandas: 1.2.4 numpy: 1.20.3 scipy: 1.6.3 netCDF4: 1.5.6 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.5.0 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: 0.9.9.0 iris: None bottleneck: None dask: 2021.04.0 distributed: 2021.04.1 matplotlib: 3.4.2 cartopy: 0.19.0.post1 seaborn: None numbagg: None pint: None setuptools: 49.6.0.post20210108 pip: 21.1.3 conda: None pytest: None IPython: 7.24.1 sphinx: None |
{ "url": "https://api.github.com/repos/pydata/xarray/issues/5604/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | 13221727 | issue |