id,node_id,number,title,user,state,locked,assignee,milestone,comments,created_at,updated_at,closed_at,author_association,active_lock_reason,draft,pull_request,body,reactions,performed_via_github_app,state_reason,repo,type
345715825,MDU6SXNzdWUzNDU3MTU4MjU=,2329,Out-of-core processing with dask not working properly?,12278765,closed,0,,,16,2018-07-30T11:19:41Z,2019-01-13T01:57:12Z,2019-01-13T01:57:12Z,NONE,,,,"Hi,

I have a bunch of GRIB files that amount to ~250GB. I want to concatenate them and save to zarr.
I concatenated them with CDO and saved to netcdf.
Now I have a ~500GB netcdf that I want to convert to zarr.
I want to convert to zarr because:
- I plan to run the analysis on a cluster, and I understand that zarr is better for that
- By using float16 and lz4 compression, I believe I can reduce the size  to ~100GB and have faster access (I think the analysis will be i/o bound).

The netcdf:
```
<xarray.Dataset>
Dimensions:  (lat: 721, lon: 1440, time: 119330)
Coordinates:
  * time     (time) datetime64[ns] 2000-01-01T06:00:00 2000-01-01T06:00:00 ...
  * lon      (lon) float32 0.0 0.25 0.5 0.75 1.0 1.25 1.5 1.75 2.0 2.25 2.5 ...
  * lat      (lat) float32 90.0 89.75 89.5 89.25 89.0 88.75 88.5 88.25 88.0 ...
Data variables:
    mtpr     (time, lat, lon) float32 ...
Attributes:
    CDI:          Climate Data Interface version ?? (http://mpimet.mpg.de/cdi)
    Conventions:  CF-1.6
    history:      Fri Jul 27 16:35:19 2018: cdo -f nc4 mergetime 2000-01.grib...
    institution:  European Centre for Medium-Range Weather Forecasts
    CDO:          Climate Data Operators version 1.9.3 (http://mpimet.mpg.de/...
```

The code:

```python
def netcdf2zarr(nc_path):
    chunks = {'time': -1, 'lat': 'auto', 'lon': 'auto'}
    encoding = {'mtpr': {'dtype': 'float16', 'compressor': zarr.Blosc(cname='lz4', clevel=9)},
                'lat': {'dtype': 'float32'},
                'lon': {'dtype': 'float32'}}
    ds = xr.open_dataset(nc_path).chunk(chunks)
    ds.to_zarr('myfile.zarr', encoding=encoding)
```

#### Problem description

I left my code to run over the weekend. After 63h of processing, the zarr store was only 1GB in size. The system monitor indicated that the Python process had 17TB worth of disk read.
At that rate, it would have taken months to finish.
Is there something that I can do to increase the processing speed?

I run Ubuntu 18.04 on a Core i7-6700 with 16GB of RAM. The disk is an HDD with a speed of ~100MB/s.

#### Output of ``xr.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.15.final.0
python-bits: 64
OS: Linux
OS-release: 4.15.0-24-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8
LOCALE: None.None

xarray: 0.10.8
pandas: 0.23.3
numpy: 1.14.2
scipy: 1.1.0
netCDF4: 1.3.1
h5netcdf: 0.6.1
h5py: 2.8.0
Nio: None
zarr: 2.2.0
bottleneck: 1.2.1
cyordereddict: 1.0.0
dask: 0.18.2
distributed: 1.22.0
matplotlib: 2.2.2
cartopy: 0.16.0
seaborn: 0.9.0
setuptools: 39.1.0
pip: 10.0.1
conda: None
pytest: None
IPython: None
sphinx: None

</details>
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/2329/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue