issue_comments: 351837521

Just to give a brief synopsis of what we are working in Dask in case it is valuable for this or other contexts, have given an overview of the relevant work below.

With Matthew's help am trying to add a keep argument to da.store. By default keep=False, which is the current behavior of da.store. If keep=True however, it returns Dask Arrays that can lazily load data written by da.store. Thus allowing the stored result to be linked to later computations before it is fully written. The compute argument of da.store affects whether to submit the storage tasks immediately (adding Futures into the resultant Dask Array) or whether to hold off until a later computation step triggers it.

This sort of functionality could be useful for a variety of situations including the one Matthew has described above. Also this could be useful for viewing partially computed results before they are totally done. Another use case could be more rapid batching of computations with many intermediate values. There is also an opportunity to re-explore caching in this context; thus, revisiting an area that many people have previously shown interest in.