html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/1784#issuecomment-367806166,https://api.github.com/repos/pydata/xarray/issues/1784,367806166,MDEyOklzc3VlQ29tbWVudDM2NzgwNjE2Ng==,1217238,2018-02-22T20:08:46Z,2018-02-22T20:08:46Z,MEMBER,"`sync=False` is a private, undocumented API. I think it's only found on the internal `to_netcdf` function in `xarray.backends.api`, not the `to_netcdf` method.
Currently, `to_netcdf()` will either return:
- `None` if a path or file-like object was passed to `to_netcdf()`.
- `bytes` representing the file's data if no path or file was provided.
In either case I think it could make sense to optionally replace the return value with a dask delayed object, which in turn evaluates to either `None` or `bytes` when computed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,282178751
https://github.com/pydata/xarray/issues/1784#issuecomment-367166682,https://api.github.com/repos/pydata/xarray/issues/1784,367166682,MDEyOklzc3VlQ29tbWVudDM2NzE2NjY4Mg==,2443309,2018-02-21T00:10:04Z,2018-02-21T00:10:04Z,MEMBER,"> What does ds.to_netcdf(...) usually return?
If `sync == False` the store is returned, otherwise nothing is returned.
> The term future, when used in a Dask context, generally refers to something that is off computing asynchronously somewhere, rather than a token that holds onto a yet-to-be-submitted lazy graph.
Thanks for the clarification. I wasn't aware of that distinction but it does make sense.
> What is store in this case?
A `store` is `AbstractWritableDataStore`, basically a wrapper class to allow us to read/write to various fileformats with various APIs under a common interface. Notably, each `store` has a `writer` attribute with a `sync` method that calls `dask.array.store`.
----
Another way to do this would be to have user code interact with the sync method directly:
```Python
store = ds.to_netcdf('file.nc', sync=False)
# store.sync calls store.writer.sync() which calls dask.array.sync
delayed_things = store.sync(compute=False)
```
This has the advantage of keeping the `to_netcdf` method a bit cleaner but does expose the `AbstractWritableDataStore` to user code which is typically not a public API object. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,282178751
https://github.com/pydata/xarray/issues/1784#issuecomment-367164232,https://api.github.com/repos/pydata/xarray/issues/1784,367164232,MDEyOklzc3VlQ29tbWVudDM2NzE2NDIzMg==,3019665,2018-02-20T23:58:47Z,2018-02-20T23:58:47Z,NONE,What is `store` in this case? Sorry not very familiar with how xarray does things.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,282178751
https://github.com/pydata/xarray/issues/1784#issuecomment-367163589,https://api.github.com/repos/pydata/xarray/issues/1784,367163589,MDEyOklzc3VlQ29tbWVudDM2NzE2MzU4OQ==,306380,2018-02-20T23:55:25Z,2018-02-20T23:55:25Z,MEMBER,"The term future, when used in a Dask context, generally refers to something that is off computing asynchronously somewhere, rather than a token that holds onto a yet-to-be-submitted lazy graph.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,282178751
https://github.com/pydata/xarray/issues/1784#issuecomment-367163456,https://api.github.com/repos/pydata/xarray/issues/1784,367163456,MDEyOklzc3VlQ29tbWVudDM2NzE2MzQ1Ng==,306380,2018-02-20T23:54:46Z,2018-02-20T23:54:46Z,MEMBER,"What does `ds.to_netcdf(...)` usually return? Typically when we specify `compute=False` we usually return a dask-like thing, often a dask.delayed object that evaluates to what *would* have been computed normally.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,282178751
https://github.com/pydata/xarray/issues/1784#issuecomment-367162456,https://api.github.com/repos/pydata/xarray/issues/1784,367162456,MDEyOklzc3VlQ29tbWVudDM2NzE2MjQ1Ng==,2443309,2018-02-20T23:49:41Z,2018-02-20T23:49:41Z,MEMBER,"@shoyer - Do you have thoughts on how this feature would present to the user? In #1811, I have added the `compute` keyword argument to `to_netcdf` and `to_zarr` and put a `futures` attribute on each store. So the workflow there would be something like:
```Python
store = ds.to_netcdf('file.nc', compute=False)
dask.compute(store.futures)
```
Before I spend too much time on #1811, I want to get some buy in on the API for this feature. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,282178751
https://github.com/pydata/xarray/issues/1784#issuecomment-352036122,https://api.github.com/repos/pydata/xarray/issues/1784,352036122,MDEyOklzc3VlQ29tbWVudDM1MjAzNjEyMg==,3019665,2017-12-15T15:38:14Z,2017-12-15T15:38:14Z,NONE,"In case anyone is curious, PR ( https://github.com/dask/dask/pull/2980 ) contains this work. Feedback welcome.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,282178751
https://github.com/pydata/xarray/issues/1784#issuecomment-351837521,https://api.github.com/repos/pydata/xarray/issues/1784,351837521,MDEyOklzc3VlQ29tbWVudDM1MTgzNzUyMQ==,3019665,2017-12-14T21:13:30Z,2017-12-14T21:13:30Z,NONE,"Just to give a brief synopsis of what we are working in Dask in case it is valuable for this or other contexts, have given an overview of the relevant work below.
With Matthew's help am trying to add a `keep` argument to `da.store`. By default `keep=False`, which is the current behavior of `da.store`. If `keep=True` however, it returns Dask Arrays that can lazily load data written by `da.store`. Thus allowing the stored result to be linked to later computations before it is fully written. The `compute` argument of `da.store` affects whether to submit the storage tasks immediately (adding `Future`s into the resultant Dask Array) or whether to hold off until a later computation step triggers it.
This sort of functionality could be useful for a variety of situations including the one Matthew has described above. Also this could be useful for viewing partially computed results before they are totally done. Another use case could be more rapid batching of computations with many intermediate values. There is also an opportunity to re-explore caching in this context; thus, revisiting an area that many people have previously shown interest in.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,282178751
https://github.com/pydata/xarray/issues/1784#issuecomment-351785902,https://api.github.com/repos/pydata/xarray/issues/1784,351785902,MDEyOklzc3VlQ29tbWVudDM1MTc4NTkwMg==,1217238,2017-12-14T17:48:48Z,2017-12-14T17:48:48Z,MEMBER,"Yes, this sounds like a natural improvement to me!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,282178751