html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/1279#issuecomment-328731021,https://api.github.com/repos/pydata/xarray/issues/1279,328731021,MDEyOklzc3VlQ29tbWVudDMyODczMTAyMQ==,2443309,2017-09-12T04:13:37Z,2017-09-12T04:13:37Z,MEMBER,see #1568 for PR that adds this,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,208903781
https://github.com/pydata/xarray/issues/1279#issuecomment-328690191,https://api.github.com/repos/pydata/xarray/issues/1279,328690191,MDEyOklzc3VlQ29tbWVudDMyODY5MDE5MQ==,2443309,2017-09-11T23:48:58Z,2017-09-12T04:13:15Z,MEMBER,"@darothen and @shoyer -
Here's a little wrapper function that does the dask and bottleneck piece...
```Python
def dask_rolling_wrapper(moving_func, a, window, min_count=None, axis=-1):
'''wrapper to apply bottleneck moving window funcs on dask arrays'''
# inputs for ghost
if axis < 0:
axis = a.ndim + axis
depth = {d: 0 for d in range(a.ndim)}
depth[axis] = window - 1
boundary = {d: np.nan for d in range(a.ndim)}
# create ghosted arrays
ag = da.ghost.ghost(a, depth=depth, boundary=boundary)
# apply rolling func
out = ag.map_blocks(moving_func, window, min_count=min_count,
axis=axis, dtype=a.dtype)
# trim array
result = da.ghost.trim_internal(out, depth)
return result
```
I don't think this would be all that difficult to drop into our current `Rolling` class.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,208903781
https://github.com/pydata/xarray/issues/1279#issuecomment-328724745,https://api.github.com/repos/pydata/xarray/issues/1279,328724745,MDEyOklzc3VlQ29tbWVudDMyODcyNDc0NQ==,2443309,2017-09-12T03:30:20Z,2017-09-12T03:30:20Z,MEMBER,@darothen - I'll open a PR in a few minutes. I'll fix the typos.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,208903781
https://github.com/pydata/xarray/issues/1279#issuecomment-328315251,https://api.github.com/repos/pydata/xarray/issues/1279,328315251,MDEyOklzc3VlQ29tbWVudDMyODMxNTI1MQ==,1217238,2017-09-10T02:24:22Z,2017-09-10T02:24:22Z,MEMBER,"@darothen Can you give an example of typical `shape` and `chunks` for your data when you load it with dask?
My sense is that we would do better to keep everything in the form of (dask) arrays, rather than converting into dataframes. For the highest performance, I would make a dask array routine that combines ghosting, map blocks and bottleneck's rolling window functions. Then it should be straightforward into rolling in place of the existing bottleneck routine.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,208903781
https://github.com/pydata/xarray/issues/1279#issuecomment-302137119,https://api.github.com/repos/pydata/xarray/issues/1279,302137119,MDEyOklzc3VlQ29tbWVudDMwMjEzNzExOQ==,1217238,2017-05-17T15:59:58Z,2017-05-17T15:59:58Z,MEMBER,"@darothen we would need to add xarray -> dask dataframe conversion functions, which don't currently exist. Otherwise I think we would still need to rewrite this (but of course the dataframe implementation could be a useful reference point).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,208903781
https://github.com/pydata/xarray/issues/1279#issuecomment-284133376,https://api.github.com/repos/pydata/xarray/issues/1279,284133376,MDEyOklzc3VlQ29tbWVudDI4NDEzMzM3Ng==,1217238,2017-03-04T07:06:25Z,2017-03-04T07:06:25Z,MEMBER,"> An idea...since we only have 1-D rolling methods in xarray, couldn't we just use map_blocks with numpy/bottleneck functions when the rolling dimension is completely contained in a dask chunk?
Yes, that would work for such cases. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,208903781
https://github.com/pydata/xarray/issues/1279#issuecomment-284132513,https://api.github.com/repos/pydata/xarray/issues/1279,284132513,MDEyOklzc3VlQ29tbWVudDI4NDEzMjUxMw==,2443309,2017-03-04T06:45:11Z,2017-03-04T06:45:11Z,MEMBER,"An idea...since we only have 1-D rolling methods in xarray, couldn't we just use `map_blocks` with numpy/bottleneck functions when the rolling dimension is completely contained in a dask chunk?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,208903781
https://github.com/pydata/xarray/issues/1279#issuecomment-281185199,https://api.github.com/repos/pydata/xarray/issues/1279,281185199,MDEyOklzc3VlQ29tbWVudDI4MTE4NTE5OQ==,1217238,2017-02-20T21:28:37Z,2017-02-20T21:28:37Z,MEMBER,"> Note that I was able to apply the rolling window by converting my variable to a pandas series with to_series(). I then could use panda's own rolling window methods. I guess that when converting to a pandas series the dask array is read in memory?
Yes, this is correct -- we automatically compute dask arrays when converting to pandas, because pandas does not have any notion of lazy arrays.
Note that we currently have two versions of rolling window operations:
1. Implemented with bottleneck. These are fast, but only work in memory. Something like ghost cells would be necessary to extend them to dask.
2. Implemented with a nested loop written in Python. These are much slower, both because of the algorithm (time O(dim_size * window_size) instead of time O(dim_size)) and implementation of the inner loop in Python instead of C, but there's no fundamental reason why they shouldn't be able to work for dask arrays basically as is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,208903781
https://github.com/pydata/xarray/issues/1279#issuecomment-281101281,https://api.github.com/repos/pydata/xarray/issues/1279,281101281,MDEyOklzc3VlQ29tbWVudDI4MTEwMTI4MQ==,1197350,2017-02-20T15:01:44Z,2017-02-20T15:01:44Z,MEMBER,It seems like the most efficient way to handle this would be to use [ghost cells](http://dask.pydata.org/en/latest/array-ghost.html) equal to the window length.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,208903781