issue_comments: 515906488

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/issues/3165#issuecomment-515906488 https://api.github.com/repos/pydata/xarray/issues/3165 515906488 MDEyOklzc3VlQ29tbWVudDUxNTkwNjQ4OA== 13084427 2019-07-29T08:55:34Z 2019-07-29T08:55:51Z NONE

Have you tried adding more chunking, e.g., along the x dimension? That’s that usual recommendation if you’re running out of memory.

Hi Shoyer,

Thanks for your reply and help. However, I have tried various chunks along each and both dimension (like 200 on x dimension, 100 on y dimension; or larger chunks like 2000 on y dimension), it doesn't work.

In both a ubuntu machine with 100 Gb memory and a local windows10 machine, it simply crashed in couple of seconds. Even though it says memory error, the code does not use much memory at all. Also even with the one dimension setup, the temp.data shows that each chunk only takes 4 mb memory (which makes me think it might be too small and then used larger chunks). I also used a new conda environment with clean install of just the necessary libraries, and the problem is still there.

Here is the neat new environment under which I tried again but gives the same errors,

Output of xr.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.7.3 | packaged by conda-forge | (default, Jul 1 2019, 21:52:21) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.15.0-51-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: None libnetcdf: None xarray: 0.12.3 pandas: 0.25.0 numpy: 1.16.4 scipy: None netCDF4: None pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: None nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: 2.1.0 distributed: 2.1.0 matplotlib: None cartopy: None seaborn: None numbagg: None setuptools: 41.0.1 pip: 19.2.1 conda: None pytest: None IPython: None sphinx: None

By the way, the above code seems to work ok with previous 0.12.1 version Xarray and bottleneck.

Cheers, Joey

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   473692721