issues: 533578755

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
533578755 MDU6SXNzdWU1MzM1Nzg3NTU= 3599 map_blocks graph construction bug 2448579 closed 0     0 2019-12-05T20:23:06Z 2019-12-07T04:30:19Z 2019-12-07T04:30:19Z MEMBER      

Just making a new issue for #3598

The tests for https://github.com/pydata/xarray/pull/3584 fail on dask == 2.8.1 with this interesting bug. Here's a reproducible example.

``` python import dask import xarray as xr

ds = xr.Dataset({'x': (('y',), dask.array.ones(10, chunks=(3,)))}) mapped = ds.map_blocks(lambda x: x) mapped.compute() # works

xr.testing.assert_equal(mapped, ds) # does not work xr.testing.assert_equal(mapped, ds.compute()) # works xr.testing.assert_equal(mapped.compute(), ds) # works xr.testing.assert_equal(mapped.compute(), ds.compute()) # works ```

The traceback is ``` ~/miniconda3/envs/dcpy/lib/python3.7/site-packages/dask/array/optimization.py in optimize(dsk, keys, fuse_keys, fast_functions, inline_functions_fast_functions, rename_fused_keys, **kwargs) 41 if isinstance(dsk, HighLevelGraph): 42 dsk = optimize_blockwise(dsk, keys=keys) ---> 43 dsk = fuse_roots(dsk, keys=keys) 44 45 # Low level task optimizations

~/miniconda3/envs/dcpy/lib/python3.7/site-packages/dask/blockwise.py in fuse_roots(graph, keys) 819 isinstance(layer, Blockwise) 820 and len(deps) > 1 --> 821 and not any(dependencies[dep] for dep in deps) # no need to fuse if 0 or 1 822 and all(len(dependents[dep]) == 1 for dep in deps) 823 ):

~/miniconda3/envs/dcpy/lib/python3.7/site-packages/dask/blockwise.py in <genexpr>(.0) 819 isinstance(layer, Blockwise) 820 and len(deps) > 1 --> 821 and not any(dependencies[dep] for dep in deps) # no need to fuse if 0 or 1 822 and all(len(dependents[dep]) == 1 for dep in deps) 823 ):

KeyError: 'lambda-6720ab0e3639d5c63fc06dfc66a3ce47-x' ```

This key is not in dependencies. From https://github.com/dask/dask/blob/67fb5363009c583c175cb577776a4f2f4e811410/dask/blockwise.py#L816-L826

python for name, layer in graph.layers.items(): deps = graph.dependencies[name] if ( isinstance(layer, Blockwise) and len(deps) > 1 and not any(dependencies[dep] for dep in deps) # no need to fuse if 0 or 1 and all(len(dependents[dep]) == 1 for dep in deps) ): new = toolz.merge(layer, *[layers[dep] for dep in deps]) new, _ = fuse(new, keys, ave_width=len(deps))

I'm not sure whether this is a bug in fuse_roots, HighLevelGraph.from_collections or in how map_blocks calls HighLevelGraph.from_collections here: https://github.com/pydata/xarray/blob/69c85b85a1ef5d13b6ca51b6b655a1f719cc5abf/xarray/core/parallel.py#L315

 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/3599/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed 13221727 issue

Links from other tables