It seems the xarray object that is sent to the workers contains a reference to the complete graph:

```python vals = da.random.random((5, 1), chunks=(1, 1)) ds = xr.Dataset({'vals': (['a', 'b'], vals)}) write = ds.to_netcdf('file2.nc', compute=False)

key = [val for val in write.dask.keys() if isinstance(val, str) and val.startswith('NetCDF')][0] wrapper = write.dask[key] len(pickle.dumps(wrapper))

14652

delayed_store = wrapper.datastore.delayed_store len(pickle.dumps(delayed_store))

14652

dask.visualize(delayed_store) ```

The size jumps to the 1.3MB if I use 500 chunks again.

The warning about the large object in the graph disappears if we delete that reference before we execute the graph: key = [val for val in write.dask.keys() if isinstance(val,str) and val.startswith('NetCDF')][0] wrapper = write.dask[key] del wrapper.datastore.delayed_store It doesn't to change the runtime though.

Ah, that seems to do the trick. I get about 4.5s for both now, and the time spent pickeling stuff is down to reasonable levels (0.022s). Also the number of function calls dropped from 1e8 to 3e5 :-)

There still seems to be some inefficiency in the pickeled graph output, I'm getting a warning about large objects in the graph:

``` /Users/adrianseyboldt/anaconda3/lib/python3.6/site-packages/distributed/worker.py:840: UserWarning: Large object of size 1.31 MB detected in task graph: ('store-03165bae-ac28-11e8-b137-56001c88cd01', <xa ... t 0x316112cc0>) Consider scattering large objects ahead of time with client.scatter to reduce scheduler burden and keep data on workers

future = client.submit(func, big_data)    # bad

big_future = client.scatter(big_data)     # good
future = client.submit(func, big_future)  # good

% (format_bytes(len(b)), s)) ```

The size scales linearly with the number of chunks (it is 13MB if there are 5000 chunks). This doesn't seem to be nearly as problematic as the original issue though.

This is after applying both #2391 and #2261.

pangeo-data/gangeo#266 sounds somewhat similar. If you increase the size of the involved arrays here, you also end up with warnings about the size of the graph: https://stackoverflow.com/questions/52039697/how-to-avoid-large-objects-in-task-graph

I haven't tried with #2261 applied, but I can try that tomorrow.

If we interpret the time spent in _thread.lock as the time the main process is waiting for the workers, then that doesn't seem to be that main problem here. We spend 60s in pickle (almost all the time), and only 7s waiting for locks. I tried looking at the contents of the graph a bit (write.dask.dicts) and compared that to the graph of the dataset itself (ds.vals.data.dask.dicts). I can't pickle those for some reason (that would be great to see where it is spending all that time), but it looks like those entries the main difference: ( <function dask.array.core.store_chunk(x, out, index, lock, return_stored)>, ( 'stack-6ab3acdaa825862b99d6dbe1c75f0392', 478 ), <xarray.backends.netCDF4_.NetCDF4ArrayWrapper at 0x32fc365c0>, (slice(478, 479, None), ), CombinedLock([<SerializableLock: 0ccceef3-44cd-41ed-947c-f7041ae280c8>, <distributed.lock.Lock object at 0x32fb058d0>]), False), I don't really know how they work, but maybe pickeling those NetCDF4ArrayWrapper objects is expensive (ie they contain a reference to something they shouldn't)?

I noticed and appreciate those plotting additions :-)

This roadmap sounds great! We plan on using xarray to store all traces in pymc4. The biggest item on my wish list for that is better support for plotting. I regularly convert traces to pandas and use the plotting functions there, or I convert it so that I can use seaborn. Better support for hierarchical indexes sounds useful as well. They can be a bit surprising right now, and they can't be serialised to netCDF. The documentation is quite important our use case. As it is, we are asking our users to learn Bayesian stats, diagnosing sampler issues and possibly some theano or tensorflow. When we switch to xarray, our users have to learn the basics of that as well. Most won't even have heard about netCDF. One thing in particular that I noticed is that many people seem to get confused about the difference between coordinates and dimensions at some point.

@jhamman Ah, I didn't see that. In hindsight the install docs seem like an obvious place to look for info about this...

@maxim-lian That pandas solution would have saved me a bit of debugging :-) I guess another option would be to only replace functions that are present in Bottleneck. As long as they only add and never change functions this would allow xarray to use even very recent additions to bottleneck.