issues: 309100522

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
309100522 MDU6SXNzdWUzMDkxMDA1MjI= 2018 MemoryError when using save_mfdataset() 1117224 closed 0     1 2018-03-27T19:22:28Z 2020-03-28T07:51:17Z 2020-03-28T07:51:17Z NONE      

Code Sample, a copy-pastable example if possible

```python import xarray as xr import dask

Dummy data that on disk is about ~200GB

da = xr.DataArray(dask.array.random.normal(0, 1, size=(12,408,1367,304,448), chunks=(1, 1, 1, 304, 448)), dims=('ensemble', 'init_time', 'fore_time', 'x', 'y'))

Perform some calculation on the dask data

da_sum = da.sum(dim='x').sum(dim='y')(2525)/(10**6)

Write to multiple files

c_e, datasets = zip(*da_sum.to_dataset(name='sic').groupby('ensemble')) paths = ['file_%s.nc' % e for e in c_e] xr.save_mfdataset(datasets, paths)

```

Problem description

Results in a MemoryError, when dask should handle writing this OOM DataArray to multiple within-memory-sized netcdf files. Related SO post here

Expected Output

12 netcdf files (grouped by the ensemble dim).

Output of xr.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.6.4.final.0 python-bits: 64 OS: Linux OS-release: 4.14.12 machine: x86_64 processor: byteorder: little LC_ALL: C LANG: C LOCALE: None.None xarray: 0.10.2 pandas: 0.22.0 numpy: 1.14.1 scipy: 1.0.0 netCDF4: 1.3.1 h5netcdf: 0.5.0 h5py: 2.7.1 Nio: None zarr: None bottleneck: 1.2.1 cyordereddict: None dask: 0.17.1 distributed: 1.21.1 matplotlib: 2.2.2 cartopy: None seaborn: 0.8.1 setuptools: 38.5.1 pip: 9.0.1 conda: None pytest: None IPython: 6.2.1 sphinx: None 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/2018/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed 13221727 issue

Links from other tables