issue_comments: 415450958

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/issues/2329#issuecomment-415450958 https://api.github.com/repos/pydata/xarray/issues/2329 415450958 MDEyOklzc3VlQ29tbWVudDQxNTQ1MDk1OA== 12278765 2018-08-23T15:02:57Z 2018-08-23T15:02:57Z NONE

It seems that I managed to get something working as it should. I first load my monthly grib files with iris, convert to xarray, then write to zarr. This uses all the CPU cores, but loads the full array into memory. Since the individual arrays are relatively small, that is not an issue. Then I load the monthly zarr stores with xarray, concatenate them with auto_combine and write to a big zarr. The memory usage peaked just above 17GB with 32 CPU threads. The array and chunks dimensions are: (time, latitude, longitude) float16 dask.array<shape=(113969, 721, 1440), chunksize=(113969, 20, 20)> I guess that reducing the chunk size will lower the memory usage.

Using that big zarr storage, plotting a map of the mean values along the time axis takes around 15min, uses all the cores and around 24GB of RAM. The strange part is: I think I tried that before and it was not working...

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   345715825