html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/1832#issuecomment-559166277,https://api.github.com/repos/pydata/xarray/issues/1832,559166277,MDEyOklzc3VlQ29tbWVudDU1OTE2NjI3Nw==,1197350,2019-11-27T16:45:14Z,2019-11-27T16:45:14Z,MEMBER,I am trying a new approach to this problem using xarray's new `map_blocks` function. See this example: https://nbviewer.jupyter.org/gist/rabernat/30e7b747f0e3583b5b776e4093266114,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,288785270
https://github.com/pydata/xarray/issues/1832#issuecomment-499645413,https://api.github.com/repos/pydata/xarray/issues/1832,499645413,MDEyOklzc3VlQ29tbWVudDQ5OTY0NTQxMw==,1197350,2019-06-06T20:01:40Z,2019-06-06T20:01:40Z,MEMBER,"In recent versions of xarray (0.12.1) and dask (0.12.1), this issue has been ameliorated significantly. I believe this issue should now be closed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,288785270
https://github.com/pydata/xarray/issues/1832#issuecomment-363482773,https://api.github.com/repos/pydata/xarray/issues/1832,363482773,MDEyOklzc3VlQ29tbWVudDM2MzQ4Mjc3Mw==,1197350,2018-02-06T16:38:31Z,2018-02-06T16:38:31Z,MEMBER,"Short answer...no luck. With the latest masters (but without the suggested dask config), I am still getting the same basic performance limitations.

I can update you more when we talk in person later today.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,288785270
https://github.com/pydata/xarray/issues/1832#issuecomment-362151486,https://api.github.com/repos/pydata/xarray/issues/1832,362151486,MDEyOklzc3VlQ29tbWVudDM2MjE1MTQ4Ng==,1197350,2018-02-01T03:59:28Z,2018-02-01T03:59:28Z,MEMBER,@mrocklin thanks for the updates. I should have some time on Friday morning to give it a try on Cheyenne.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,288785270
https://github.com/pydata/xarray/issues/1832#issuecomment-358106768,https://api.github.com/repos/pydata/xarray/issues/1832,358106768,MDEyOklzc3VlQ29tbWVudDM1ODEwNjc2OA==,1197350,2018-01-16T21:10:37Z,2018-01-16T21:10:37Z,MEMBER,Or maybe real data just gets in the way of the core dask issue?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,288785270
https://github.com/pydata/xarray/issues/1832#issuecomment-358106627,https://api.github.com/repos/pydata/xarray/issues/1832,358106627,MDEyOklzc3VlQ29tbWVudDM1ODEwNjYyNw==,1197350,2018-01-16T21:10:12Z,2018-01-16T21:10:12Z,MEMBER,I am developing a use case for this scenario using real data. I will put the data in cloud storage as soon as #1800 is merged. That should make it easier to debug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,288785270
https://github.com/pydata/xarray/issues/1832#issuecomment-358017392,https://api.github.com/repos/pydata/xarray/issues/1832,358017392,MDEyOklzc3VlQ29tbWVudDM1ODAxNzM5Mg==,1197350,2018-01-16T16:21:37Z,2018-01-16T16:21:37Z,MEMBER,"Below is how I work around the issue in practice: writing a loop over each item in the groupby, and then looping over each variable, loading, and writing to disk.

```python
gb = ds.groupby('time.month')
for month, dsm in gb:
    dsm_anom2 = ((dsm - ds_mm.sel(month=month))**2).mean(dim='time')
    dsm_anom2 = dsm_anom2.rename({f: f + '2' for f in fields})
    dsm_anom2.coords['month'] = month
    for var in dsm_anom2.data_vars:
        filename = save_dir + '%02d.%s_%s.nc' % (month, prefix, var)
        print(filename)
        ds_out = dsm_anom2[[var]].load()
        ds_out.to_netcdf(filename)
```

Needless to say, this feels more like my pre-xarray/dask workflow.

Since @mrocklin has made it pretty clear that dask will not automatically solve this for us any time soon, we need to brainstorm some creative ways to make this extremely common use case more friendly with out-of-core data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,288785270
https://github.com/pydata/xarray/issues/1832#issuecomment-358003205,https://api.github.com/repos/pydata/xarray/issues/1832,358003205,MDEyOklzc3VlQ29tbWVudDM1ODAwMzIwNQ==,1197350,2018-01-16T15:41:25Z,2018-01-16T15:41:25Z,MEMBER,"The operation
```python
ds_anom = ds - ds.mean(dim='time')
```
is also extremely common. Both should work well by default.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,288785270