issue_comments

I think this triggers a case that dask's scheduler doesn't handle well, related to this issue: https://github.com/dask/dask/issues/874

Actually, there does seem to be something fishy going on here. I find that I'm able to execute temp.rolling(x=100).construct('window').mean('window').compute() successfully but not temp.rolling(x=100).mean().compute(), even though that should mostly be equivalent to the former.

For context, xarray's rolling window code creates a "virtual dimension" for the rolling window. So if your chunks are size (5000, 100) before the rolling window, they are size (5000, 100, 100) within the rolling window computation. So it's not entirely surprising that there are more issues with memory usage -- these are much bigger arrays, e.g., see ```

temp.rolling(x=100).construct('window') <xarray.DataArray (x: 5000, y: 50000, window: 100)> dask.array<shape=(5000, 50000, 100), dtype=float64, chunksize=(50, 100, 100)> Dimensions without coordinates: x, y, window ```

You want to use the chunks argument inside da.zeros, e.g., da.zeros((5000, 50000), chunks=100).

On Mon, Jul 29, 2019 at 3:30 PM peterhob notifications@github.com wrote:

Did you try converting np.zeros((5000, 50000) to use dask.array.zeros instead? The former will allocate 2 GB of data within each chunk

Thank you for your suggestion. Tried as you suggested, still with same error.

import numpy as npimport xarray as xrimport dask.array as da# from dask.distributed import Client temp= xr.DataArray(da.zeros((5000, 50000)),dims=("x","y")).chunk({"y":100, }) temp.rolling(x=100).mean()

I have also tried saving the array to nc file and read it after that. Still rolling gives same error (with or without bottleneck and different chunks). Even though it says memory error, it doesn't consume too much memory.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/3165?email_source=notifications&email_token=AAJJFVT7OTCOARO4WQZBCODQB5VQ5A5CNFSM4IHLGUAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3CGFKY#issuecomment-516186795, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJJFVXZRBM7FOHY4NAT53TQB5VQ5ANCNFSM4IHLGUAA .

Did you try converting np.zeros((5000, 50000) to use dask.array.zeros instead? The former will allocate 2 GB of data within each chunk

Have you tried adding more chunking, e.g., along the x dimension? That’s that usual recommendation if you’re running out of memory.