issue_comments

Looking at the task graph my first guess is that @shoyer is correct, and that we've found another case that the scheduler should be able to handle well, but doesn't. This hasn't happened in a while, but it always leads to improvements whenever we find such a problem.

For a case this complex I think we either need to reduce it to a particular graph motif on which we schedule poorly or we first need to develop a better way to visualize traces of the scheduler's behavior. I've started a separate dask issue: https://github.com/dask/dask/issues/994

For the near future I don't have a solution to @Scheibs 's research problem (sorry!) This will probably require tweaking dask scheduler internals which probably won't happen by me in the next couple of weeks. I'm very happy that people brought this to my attention though.

Slowly taking a look at this now. Large PDF for the full computation, if anyone is interested:

dask.pdf

Downloaded

My apologies for the slow response (very busy week, lots of exciting stuff, sadly results in poor user response).

I can access that page easily but the download seems to halt after 11MB

mrocklin continuum io

On Mon, Feb 8, 2016 at 1:10 AM, Scheibs notifications@github.com wrote:

@mrocklin https://github.com/mrocklin I put my folder on a ftp server, can I have an email address to send you the login information ?

@shoyer https://github.com/shoyer I have tried replacing sum() by dot() but I get an error "Data Array has no attribute 'dot' "

— Reply to this email directly or view it on GitHub https://github.com/pydata/xarray/issues/729#issuecomment-181265026.

I would generally send such a large file by hosting it at a web-accessible location. Perhaps you are at an institution were you have access to host files online?

Sorry for the delay in response.

Nothing here seems dangerous to me. @shoyer does the writeup above raise any questions for you?

If convenient, it would be interesting to see the output of a few of the dask profilers:

$ conda install bokeh $ pip install cachey

``` python import cachey from dask.diagnostics import CacheProfiler, ResourceProfiler, Profiler, visualize with Profiler() as prof, CacheProfiler(metric=cachey.nbytes) as cprof, ResourceProfiler() as rprof: # call the final dataset.to_netcdf() function

visualize([prof, cprof, rprof], file_path='profile.html') ```

And then upload that file somewhere, perhaps to a gist. In order to make this run to completion you might have to operate on a subset of the dataset.

Alternatively, is there a way for me to recreate a version of this dataset on my local machine? @shoyer is there a way to capture the metadata of netcdf files and reinstantiate empty copies of them on another machine?

@Scheibs can you try calling these lines to remove multi-threading and see if the problem persists?

python import dask dask.set_options(get=dask.async.get_sync)

I agree with @shoyer that it would be very useful to see what you're doing that causes this problem.