issue_comments

The problem was only with very large arrays, so difficult to reproduce here.

Fortunately, when I just now updated to the very latest xarray/dask, the problem seems to have gone away.

I was puzzled because things worked fine -- as expected -- but only for smaller arrays. I will close this issue.

Something is very wrong here. is this reproducible?

I use http://xarray.pydata.org/en/stable/generated/xarray.apply_ufunc.html because it is faster.

El mié., 20 may. 2020 a las 20:01, Javier Ruano (javier.ruanno@gmail.com) escribió:

if you append compute() it should not be a lazy operation. But my advice is like user only.

El mié., 20 may. 2020 a las 19:51, Rob Hetland (notifications@github.com) escribió:

I have a large DataSet, including these DataArrays:

<xarray.DataArray 'temp' (ocean_time: 1325, s_rho: 30, eta_rho: 602, xi_rho: 677)> dask.array<concatenate, shape=(1325, 30, 602, 677), dtype=float32, chunksize=(1, 1, 602, 677)>

and

<xarray.DataArray 'zeta' (ocean_time: 1325, eta_rho: 602, xi_rho: 677)> dask.array<concatenate, shape=(1325, 602, 677), dtype=float32, chunksize=(1, 602, 677)>

(The coordinates and attributes excluded for brevity, but they match in the right ways.)

When I do math operations with the 4D DataArray (temp) and 3D DataArray (zeta), no problem:

ds.zeta * ds.temp

<xarray.DataArray (ocean_time: 1325, eta_rho: 602, xi_rho: 677, s_rho: 30)> dask.array<mul, shape=(1325, 602, 677, 30), dtype=float32, chunksize=(1, 602, 677, 1)>

This returns an object instantly, and the result is lazily evaluated. However, if I just try to add temp to itself,

ds.temp + ds.temp

this fails (eventually) as my medium sized computer runs out of memory, since it starts to evaluate the numbers as if I did a compute() or asked for the values. Note 2ds.temp or ds.temp*2 is lazily evaluated, and returns an object instantly. Chunk size does not seem to be an issue, as I have tried a number of reasonable choices without success.

Why can't such simple math operations between two large arrays also be lazily evaluated?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/4085, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIGDFO2ZUJJOVNCN4JRJYDTRSQKBTANCNFSM4NGFVQRQ .

if you append compute() it should not be a lazy operation. But my advice is like user only.

El mié., 20 may. 2020 a las 19:51, Rob Hetland (notifications@github.com) escribió:

I have a large DataSet, including these DataArrays:

<xarray.DataArray 'temp' (ocean_time: 1325, s_rho: 30, eta_rho: 602, xi_rho: 677)> dask.array<concatenate, shape=(1325, 30, 602, 677), dtype=float32, chunksize=(1, 1, 602, 677)>

and

<xarray.DataArray 'zeta' (ocean_time: 1325, eta_rho: 602, xi_rho: 677)> dask.array<concatenate, shape=(1325, 602, 677), dtype=float32, chunksize=(1, 602, 677)>

(The coordinates and attributes excluded for brevity, but they match in the right ways.)

When I do math operations with the 4D DataArray (temp) and 3D DataArray (zeta), no problem:

ds.zeta * ds.temp

<xarray.DataArray (ocean_time: 1325, eta_rho: 602, xi_rho: 677, s_rho: 30)> dask.array<mul, shape=(1325, 602, 677, 30), dtype=float32, chunksize=(1, 602, 677, 1)>

This returns an object instantly, and the result is lazily evaluated. However, if I just try to add temp to itself,

ds.temp + ds.temp

this fails (eventually) as my medium sized computer runs out of memory, since it starts to evaluate the numbers as if I did a compute() or asked for the values. Note 2ds.temp or ds.temp*2 is lazily evaluated, and returns an object instantly. Chunk size does not seem to be an issue, as I have tried a number of reasonable choices without success.

Why can't such simple math operations between two large arrays also be lazily evaluated?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/4085, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIGDFO2ZUJJOVNCN4JRJYDTRSQKBTANCNFSM4NGFVQRQ .