issue_comments: 286497255

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/issues/1308#issuecomment-286497255 https://api.github.com/repos/pydata/xarray/issues/1308 286497255 MDEyOklzc3VlQ29tbWVudDI4NjQ5NzI1NQ== 7300413 2017-03-14T17:27:06Z 2017-03-14T17:31:32Z NONE

Hello Stephan,

The shape of the full data, if I read from within xarray, is (time, level, lat, lon), with level=60, lat=41, lon=480. time is 4*365*7 ~ 10000.

I am chunking only along longitude, using lon=100. I previously chunked along time, but that used too much memory (~45GB out of 128 GB) since the data is split into one file per month, and reading annual data would require reading many files into memory.

Superficially, I would think that both of the above would take similar amounts of time. In fact, calculating a daily climatology also requires grouping the four 6 hourly data points into a single day as well, which seems to be more complicated. However, it seems to run faster!

Thanks, Joy

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   214088387