issue_comments

Just looking at a task manager while a task executes can give you a sense of what's going on. Dask also has some diagnostics that may be helpful: http://dask.pydata.org/en/latest/diagnostics.html On Wed, Jul 20, 2016 at 11:44 AM Saulo Meirelles notifications@github.com wrote:

No, not really. I got no error message whatsoever. Is there any test I can do to tackle this?

Sent from Smartphone. Please forgive typos.

On Jul 20, 2016 8:41 PM, "Stephan Hoyer" notifications@github.com wrote:

I decided to wait for .load() to do the job but the kernel dies after a while.

Are you running out of memory? Can you tell what's going on? This is a little surprising to me.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/912#issuecomment-234042142, or mute the thread < https://github.com/notifications/unsubscribe-auth/AHKCTXaBpbA0ieSdI2I_hIUjVBxuKaNeks5qXmvPgaJpZM4JQ0_D

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/912#issuecomment-234043292, or mute the thread https://github.com/notifications/unsubscribe-auth/ABKS1ujXItyYDLgA4ZtBkHEbTBTiTnrvks5qXmylgaJpZM4JQ0_D .

I decided to wait for .load() to do the job but the kernel dies after a while.

Are you running out of memory? Can you tell what's going on? This is a little surprising to me.

It's worth noting that conc_avg = ds.conc_profs.chunk({'burst': 10}).mean(('z','duration')) doesn't actually do any computation -- that's why it's so fast. It just sets up the computation graph. No computation happens until you write .load().

When you write ds.conc_profs.chunk(2400), it sets up the data to be loaded in a giant chunk, almost the entire file at once. Even if you use .isel() afterwards, dask does not always manage to subset the data from the initial chunk. (Sometimes it does succeed, which makes this a little confusing.)

You will probably be more successful if you try something like ds.conc_profs.chunk({'burst': 10}) instead, which keeps the intermediate chunks to a reasonable size.

Thanks for describing that -- I misread your initial description and thought you were using open_mfdataset rather than open_dataset (the glob threw me off!). The source of these files shouldn't matter once you have it in a netCDF file.

What do the original input files look like, before you join them together? This may be a case where the dask.array task scheduler does very poorly.