issue_comments

Tank you for your help on this, I will try myself to improve my knoledge of xarray and dask ;)

@mrocklin is that new link working ? Because it's gonna be closed tomorow.

You don't have to apologize for the delay, i'm already grateful for your time on this !

I don't understand why this download won't work with you; I tried with other and everything was ok... Maybe you can try with this link :
https://sharing.oodrive.com/easyshare/fwd/link=BS6TALAs9CiQg.WNotfITA

@mrocklin I finally gave you a free acces path on ftp://ftp.irsn.fr/argon/SHARE/

@mrocklin I put my files on ftp://ftp.irsn.fr/argon/SHARE/, everything is in the DEBUG.zip file.

@shoyer I have tried replacing sum() by dot() but I get an error "Data Array has no attribute 'dot' "

@shoyer @mrocklin I can send you my files, with the two netCDF4 source files and the script. I will also join the Bokeh graph for only one variable which fit with memory. Do you know how I can send you the ZIP folder ? It's 830 MB.

@mrocklin I have tried your lines, it seemed to work but finally I got the memory error message. However the crash was different than what I used to have...

My dataset is a set of several variables, which are calculated like this :

I have DataArray of unitary data for four rainfall height(0,5,10,15 mm/days,"SYMB") and another DataArray with spatialized rainfall height values ("RAIN").

SYMB <xray.DataArray (Hpluie: 4, time: 203, NIsoSource: 56, Denree: 19, Paliers: 23)> dask.array<xray-Fo..., shape=(4, 203, 56, 19, 23), dtype=float64, chunksize=(4, 50, 10, 19, 5)>

RAIN <xray.DataArray (Paliers: 23, DimK0: 1, DimJ0: 37, DimI0: 15)> dask.array<xray-Ra..., shape=(23, 1, 37, 15), dtype=float64, chunksize=(5, 1, 20, 15)>

Then I created an array which is the result of interpolation between this two arrays The function is :

``` python def interp(array,pluie):

p0 = (array.sel(Hpluie = 5) -array.sel(Hpluie = 0))/5
p1 = (array.sel(Hpluie = 10) -array.sel(Hpluie = 5))/5
p2 = (array.sel(Hpluie = 15) -array.sel(Hpluie = 10))/5

interp = (p0*(pluie.where(pluie<5) - array.Hpluie.sel(Hpluie=5)) +array.sel(Hpluie=5)).fillna(0)\
    +(p1*(pluie.where((pluie>=5) & (pluie<=10)) - array.Hpluie.sel(Hpluie=5)) +array.sel(Hpluie=5)).fillna(0)\
    +(p2*(pluie.where(pluie>10) - array.Hpluie.sel(Hpluie=10)) + array.sel(Hpluie=10)).fillna(0)   
return interp

```

So M = interp(SYMB,RAIN)

M <xray.DataArray (time: 203, NIsoSource: 56, Denree: 19, Paliers: 23, DimK0: 1, DimJ0: 37, DimI0: 15)> dask.array<elemwis..., shape=(203, 56, 19, 23, 1, 37, 15), dtype=float64, chunksize=(50, 10, 19, 5, 1, 20, 15)>

My final variable is the product of this interpolated array and an other spatialized variable("FLUX")

FLUX <xray.DataArray 'Flux_Moyen_Depot_Sec' (Paliers: 23, DimK0: 1, DimJ0: 37, DimI0: 15, NIsoSource: 56)> dask.array<xray-Fl..., shape=(23, 1, 37, 15, 56), dtype=float64, chunksize=(5, 1, 20, 15, 10)>

RES = (M * FLUX).sum("Paliers")

RES = <xray.DataArray (DimK0: 1, DimJ0: 37, DimI0: 15, NIsoSource: 56, time: 203, Denree: 19)> dask.array<p_reduc..., shape=(1, 37, 15, 56, 203, 19), dtype=float64, chunksize=(1, 20, 15, 10, 50, 19

I hope this is at less understandable... thanks a lot for your help