issue_comments

Error still present in 0.14.0

I believe the bug occurs in dask_array_ops.py: rolling_window

My best guess at understanding the code: I believe there is an attempt to "pad" rolling windows to ensure the rolling windows doesn't miss data across chunk boundaries. I think the "padding" is supposed to be truncated later, but something is miscalculated and the final array ends up with the wrong chunking.

In the case I presented, the "chunking" happens along a different dimension to the "rolling" and padding is not necessary. Perhaps something goes haywire because the code was written to guard against rolling along a chunked dimension (and missing data across chunk boundaries)? Additionally, as the padding is not necessary in this case, there is a performance penalty that could be avoided?

A simple fix for my case is to not do any "padding" whenever the chunksize along the rolling dimension is equal to the arraysize along the rolling dimension.

e.g. in the function dask_array_ops.py: rolling_window

pad_size = max(start, end) + offset - depth[axis]

becomes

if a.shape[axis] == a.chunksize[axis]:
    pad_size = 0
else:
    pad_size = max(start, end) + offset - depth[axis]

This fixes the code for my usage case, perhaps someone could advise if I have understood the issue correctly?

Using xarray=0.13.0, the chunking behaviour has changed again (but still incorrect): ``` <xarray.DataArray (item: 300, day: 7653)> dask.array<xarray-\<this-array>, shape=(300, 7653), dtype=float64, chunksize=(20, 7653), chunktype=numpy.ndarray> Coordinates: * item (item) object '0' '1' '2' '3' '4' ... '295' '296' '297' '298' '299' * day (day) datetime64[ns] 1990-01-01 1990-01-02 ... 2019-05-01

<xarray.DataArray (item: 300, day: 7653)> dask.array<where, shape=(300, 7653), dtype=float64, chunksize=(20, 7648), chunktype=numpy.ndarray> Coordinates: * item (item) object '0' '1' '2' '3' '4' ... '295' '296' '297' '298' '299' * day (day) datetime64[ns] 1990-01-01 1990-01-02 ... 2019-05-01

ValueError
... ```

Note the chunksize is now (20, 7648) instead of (20, 7653).

I believe it might be related to https://github.com/pydata/xarray/pull/2942 -> I think disabling bottleneck for dask arrays for the rolling operation caused the bug above to appear (so the bug may have been there for a while, but doesn't appear because bottleneck was being used).

Doing a quick trace, in rolling.py , I think it's the line windows = self.construct(rolling_dim) in the reduce function which creates windows with incorrect chunking, possibly as a consequence of some transpose operations and dimension mix up?

It seems strange that other applications aren't having problems with this, unless I am doing something different in my code? Note that I am very specifically chunking in a different dimension to the rolling operation.

Confirm bug is still present in 0.13.0.

Some additional notes:

The bug also appears for xarray=0.12.2 (so I presume was introduced between 0.12.1 and 0.12.2). Other rolling operations are similarly affected - replacing .mean() in the sample code with .count(), .sum(), .std(), .max() etc results in the same erroneous chunking behaviour.

Another workaround is to downgrade xarray to 0.12.1