issue_comments

In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity

If this issue remains relevant, please comment here or remove the stale label; otherwise it will be marked as closed automatically

Also, the following code seems to accomplish the same as the above:

``` def apply_func_rolling(func, args, kwargs): # determine rolling parameters, and remove them from kwargs apply_func_kwargs = {'input_core_dims', 'output_core_dims', 'vectorize', 'join', 'dataset_join', 'keep_attrs', 'exclude_dims', 'dataset_fill_value', 'kwargs', 'dask', 'output_dtypes', 'output_sizes'} min_periods = kwargs.pop('min_periods', None) center = kwargs.pop('center', False) dim = xr.core.utils.either_dict_or_kwargs(kwargs.pop('dim', None), {k: v for k, v in kwargs.items() if k not in apply_func_kwargs}, 'apply_func_rolling') if len(dim) != 1: raise ValueError("precisely one rolling dimension must be specified") rolling_dim = list(dim.keys())[0] kwargs.pop(rolling_dim) temp_rolling_dim = 'temp{}__'.format(rolling_dim) # change input_core_dims rolling_dim values to temp_rolling_dim input_core_dims = kwargs.get('input_core_dims', None) if input_core_dims: kwargs['input_core_dims'] = [[(temp_rolling_dim if (dim_ == rolling_dim) else dim_) for dim_ in dims_] for dims_ in input_core_dims] # change exclude_dims rolling_dim values to temp_rolling_dim exclude_dims = kwargs.get('exclude_dims', None) if exclude_dims: kwargs['exclude_dims'] = [[(temp_rolling_dim if (dim_ == rolling_dim) else dim_) for dim_ in dims_] for dims_ in exclude_dims] # call apply_func() with rolling-constructed objects return xr.apply_ufunc(func, [(arg.rolling(dim=dim, min_periods=min_periods, center=center). construct(temp_rolling_dim) if (rolling_dim in arg.dims) else arg) for arg in args], **kwargs)

apply_func_rolling(lambda a, b, w: ..., variables, observations, weights, date=N, input_core_dims=[['date', 'dim1', 'dim2', 'var'], ['date', 'dim1', 'dim2'], ['date', 'dim1', 'dim2']], output_core_dims=[['var']], vectorize=True) ```

construct method does not allocate that large array in memory. It uses the strided trick and therefore consumers only the order of 1000x1000x1000.

Ah. I didn't realize that. Good to know.

What I'm actually looking to do is a rolling weighted regression. I have three DataArrays: - observations, dims=('date', 'dim1', 'dim2') - variables, dims=('date', 'dim1', 'dim2', 'var') - weights, dims=('date', 'dim1', 'dim2')

I want to calculate a regression_coefficients DataArray with dims=('date', 'var'), where for each date it has the weighted regression coefficients calculated over the trailing N dates (over 'dim1' and 'dim2'). One way would be to put the three DataArrays in a Dataset, and then use a newly-defined Dataset.rolling().apply(). Another way would be to use an enhanced version of apply_ufunc() that can take Rolling objects. But now that I know that DataArrayRolling.construct() won't kill my machine, I'll try apply_ufunc() with the three DataArrayRolling.construct() objects. I'd welcome other suggestions.

OK, I seem to have got my problem working using: apply_ufunc(lambda a, b, w: ..., variables.rolling(date=N).construct('temp_date'), observations.rolling(date=N).construct('temp_date'), weights.rolling(date=N).construct('temp_date'), input_core_dims=[['temp_date', 'dim1', 'dim2', 'var'], ['temp_date', 'dim1', 'dim2'], ['temp_date', 'dim1', 'dim2']], output_core_dims=[['var']], vectorize=True) Still, I wonder if there isn't a more "natural" way of accomplishing this.

E.g. consider a 1000x1000x1000 array for which I want to apply rolling window of length 500 along the final dimension; I believe Rolling.construct/reduce will construct a 1000x1000x1000x500 array.

construct method does not allocate that large array in memory. It uses the strided trick and therefore consumers only the order of 1000x1000x1000. But sometimes the user function does not like strangely strided array, e.g. nansum allocate large array with 1000x1000x1000x500 internally.

What function are you using?

Maybe we can add apply method but I think we need to decide a good balance between memory and computational efficiency.

I think there are actually a couple different ways Rolling.apply could work, but this seems like one possible way:

``` from xarray.core.utils import maybe_wrap_array from xarray.core.combine import concat

def rolling_apply(rolling, func, args, kwargs): applied = [maybe_wrap_array(label, func(arr, args, **kwargs)) for label, arr in rolling] combined = concat(applied, dim=rolling.obj.coords[rolling.dim]) return combined ```

The problem I have with Rolling.construct is the same that I have with Rolling.reduce: it's very (potentially) memory-inefficient. E.g. consider a 1000x1000x1000 array for which I want to apply rolling window of length 500 along the final dimension; I believe Rolling.construct/reduce will construct a 1000x1000x1000x500 array. This can quickly get out of hand.

Separately, maybe apply_ufunc should accepts Rolling objects?

Maybe we need a better API similar to Groupby.apply, but it can be doable by construct method that gives another DataArray with windowed dimension attached to the last position.

We can apply any function on this constructed DataArray.

I think what is needed are DataArrayRolling.apply and DatasetRolling.apply (like DataArrayGroupBy.apply and DatasetGroupBy.apply). The problem with the reduce methods is that they are memory-inefficient.

I'd love to, I can take a look at the code, but I'm afraid my skills are not up to the task yet... How would one implement slicing and indexing? By adding a method to DataArrayRolling?

@ziofil - We don't support indexing on the rolling iterator at the moment. You could just grab the windows you want in this way:

```Python w1 = 2 w2 = 7 for i, (k, v) in enumerate(r): if i > w1 and i <= w2: print(k.item(), v.values)

3 [2 3] 4 [3 4] 5 [4 5] 6 [5 6] 7 [6 7] ```

By the way, I think we'd be happy to see some feature development on this front if you are interested in developing the rolling objects out further.

I didn't know I could just iterate through it! Thank you. It's very fast too. One thing though: I can iterate through the whole thing, but as it doesn't support indexing and it's not subscriptable, how do I extract the first n windows? Or from window w1 to window w2?

@shoyer - this is really odd. Its probably coming from https://github.com/pydata/xarray/blob/master/xarray/core/rolling.py#L149-L150 but I don't remember exactly what/why we're trying to accomplish here.

You can iterate through a rolling object as a generator, e.g., In [6]: for k, v in r: ...: print(k.item(), v.values) ...: 0 [ True] 1 [0 1] 2 [1 2] 3 [2 3] 4 [3 4] 5 [4 5] 6 [5 6] 7 [6 7] 8 [7 8] 9 [8 9] Does this help?

Note that something weird seems to be going on with the first time window. I don't know why it's a boolean. @jhamman any idea what's going on?