issue_comments

No, not really. I got no error message whatsoever. Is there any test I can do to tackle this?

Sent from Smartphone. Please forgive typos.

On Jul 20, 2016 8:41 PM, "Stephan Hoyer" notifications@github.com wrote:

I decided to wait for .load() to do the job but the kernel dies after a while.

Are you running out of memory? Can you tell what's going on? This is a little surprising to me.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/912#issuecomment-234042142, or mute the thread https://github.com/notifications/unsubscribe-auth/AHKCTXaBpbA0ieSdI2I_hIUjVBxuKaNeks5qXmvPgaJpZM4JQ0_D .

True.

I decided to wait for .load() to do the job but the kernel dies after a while.

Thanks, @shoyer !

Setting smaller chunks helps, however my issue is the way back.

This is fine:

%time conc_avg = ds.conc_profs.chunk({'burst': 10}).mean(('z','duration'))

CPU times: user 24 ms, sys: 0 ns, total: 24 ms Wall time: 23.8 ms

But this:

%time result = conc_avg.load()

takes an insane amount of time which intrigues me because is just a vector with 2845 points.

Is there another way to tackle this without dask like using a for-loop?

If dask is the way to go, what would be the quickest way to convert to numpy array?

I've tried to create individual nc-files and then read them all using open_mfdataset but I got an error for opening too many files which was reported here before.

The glob is just a (bad) habit because I normally read multiple files. O_0

Cheers,

The input files are 2485 nested mat-files that come out from a measurement device. I read them in Python ( loadmat(matfile) ) and turn them into numpy arrays like this:

``` matfiles = glob('*sed.mat')

matfiles = sorted(matfiles ,key=lambda x: extract_number(x) )


if matfiles:

    ts = 2400
    zs = 160

    Burst        = np.empty(len(matfiles))
    Time         = np.empty((ts,len(matfiles)), dtype='datetime64[s]')
    ConcProf     = np.empty((ts,zs,len(matfiles)), dtype='float64')
    GsizeProf    = np.empty((ts,zs,len(matfiles)), dtype='float64')

```

Afterwards, I populate the matrices in a loop:

``` def f(i):
Dist, Burst[i], Time[:,i], ConcProf[:,:,i], GsizeProf[:,:,i] = getABSpars(matfiles[i])

```

where

``` def getABSpars(matfile):

ndata = loadmat(matfile)

Dist  = ndata['r']

t_dic = ndata['BurstInfo']['StartTime']

try:
    t_dt  = dt.datetime.strptime(t_dic, '%d-%b-%Y %H:%M:%S')
except:
    t_dic = t_dic + ' 00:00:00'
    t_dt  = dt.datetime.strptime(t_dic, '%d-%b-%Y %H:%M:%S')

t_range   = date_range( t_dt,
            periods = ndata['MassProfiles'].shape[1],
            freq    = ndata['BurstInfo']['MassProfileInterval']+'L')

Burst         = int(ndata['BurstInfo']['BurstNumber'])
Time          = t_range
ConcProf      = np.asarray(ndata['MassProfiles'] ).T
GsizeProf     = np.asarray(ndata['SizeProfiles']*1e6).T

return Dist, Burst, Time, ConcProf, GsizeProf

```

Using the multiprocessing package:

pool = ThreadPool(4) pool.map(f, range(len(matfiles))) pool.close()

Finally I create the xarray dataset and then save into a nc-file:

``` ds = xray.Dataset( { 'conc_profs' : ( ['duration', 'z', 'burst'], ConcProf ), 'grainSize_profs' : ( ['duration', 'z', 'burst'], GsizeProf ), 'burst_duration' : ( ['duration'], np.linspace(0,299, Time.shape[0]) ), }, coords = {'time' : (['duration', 'burst'], Time) , 'zdist' : (['z'], Dist), 'burst_nr' : (['burst'], Burst) } )

ds.to_netcdf('ABS_conc_size_12m.nc' , mode='w')

```

It costs me around 1 h to generate the nc-file.

Could this be the reason of my headaches?

Thanks!