home / github / issue_comments

Menu
  • GraphQL API
  • Search all tables

issue_comments: 631636660

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/issues/4085#issuecomment-631636660 https://api.github.com/repos/pydata/xarray/issues/4085 631636660 MDEyOklzc3VlQ29tbWVudDYzMTYzNjY2MA== 34353851 2020-05-20T18:07:22Z 2020-05-20T18:07:22Z NONE

I use http://xarray.pydata.org/en/stable/generated/xarray.apply_ufunc.html because it is faster.

El mié., 20 may. 2020 a las 20:01, Javier Ruano (javier.ruanno@gmail.com) escribió:

if you append compute() it should not be a lazy operation. But my advice is like user only.

El mié., 20 may. 2020 a las 19:51, Rob Hetland (notifications@github.com) escribió:

I have a large DataSet, including these DataArrays:

<xarray.DataArray 'temp' (ocean_time: 1325, s_rho: 30, eta_rho: 602, xi_rho: 677)> dask.array<concatenate, shape=(1325, 30, 602, 677), dtype=float32, chunksize=(1, 1, 602, 677)>

and

<xarray.DataArray 'zeta' (ocean_time: 1325, eta_rho: 602, xi_rho: 677)> dask.array<concatenate, shape=(1325, 602, 677), dtype=float32, chunksize=(1, 602, 677)>

(The coordinates and attributes excluded for brevity, but they match in the right ways.)

When I do math operations with the 4D DataArray (temp) and 3D DataArray (zeta), no problem:

ds.zeta * ds.temp

<xarray.DataArray (ocean_time: 1325, eta_rho: 602, xi_rho: 677, s_rho: 30)> dask.array<mul, shape=(1325, 602, 677, 30), dtype=float32, chunksize=(1, 602, 677, 1)>

This returns an object instantly, and the result is lazily evaluated. However, if I just try to add temp to itself,

ds.temp + ds.temp

this fails (eventually) as my medium sized computer runs out of memory, since it starts to evaluate the numbers as if I did a compute() or asked for the values. Note 2ds.temp or ds.temp*2 is lazily evaluated, and returns an object instantly. Chunk size does not seem to be an issue, as I have tried a number of reasonable choices without success.

Why can't such simple math operations between two large arrays also be lazily evaluated?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/4085, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIGDFO2ZUJJOVNCN4JRJYDTRSQKBTANCNFSM4NGFVQRQ .

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  621968474
Powered by Datasette · Queries took 0.678ms · About: xarray-datasette