issues

What is your issue?

I'm doing a lengthy computation, which involves hundreds of GB of data using chunks and map_blocks() so that things fit into RAM and can be done in parallel. From time to time, the following error is raised:

pandas.errors.InvalidIndexError: Reindexing only valid with uniquely valued Index objects

The line where this takes place looks pretty harmless:

x = a * b.sel(c=d.c)

It's a line inside the function func which is passed to a map_blocks() call. In this case a and b are xr.DataArray or xr.DataSet objects shadowed from outer scope and d is the parameter obj for map_blocks().

That means, the line below in the traceback looks like this:

xr.map_blocks(
    lambda d: worker(d).compute().chunk({"time": None}),
    d,
    template=template)

I guess it's some kind of race condition, since it's not 100% reproducible, but I have no idea how to further investigate the issue to create a proper bug report or fix my code.

Do you have any hint how I could continue building a minimal example or so in such a case? What does the error message want to tell me?

Stumbled upon a unit test which didn't test anything.

Problem description

dask 1.1.0 moved atop() to blockwise() and introduced a warning when atop() is used.

Related

• upstream ticket and PR of dask change: dask/dask#4348 dask/dask#4035
• the warning in the dask documentation in an xarray example, probably not on purpose
• warnings have been already discussed in #2727, but not fixed there
• same issue in a different project: pytroll/satpy#608

Code Sample

```python import numpy as np import xarray as xr

xr.DataArray(np.ones(1000)) d = xr.DataArray(np.ones(1000)) d.to_netcdf('/tmp/ones.nc') d = xr.open_dataarray('/tmp/ones.nc', chunks=10) xr.apply_ufunc(lambda x: 42 * x, d, dask='parallelized', output_dtypes=[np.float64]) ```

This outputs the warning: ...lib/python3.7/site-packages/dask/array/blockwise.py:204: UserWarning: The da.atop function has moved to da.blockwise warnings.warn("The da.atop function has moved to da.blockwise")

Expected Output

No warning. As user of a recent version of dask and xarray, there shouldn't be any warnings if everything is done right. The warning should be tackled inside xarray somehow.

Solution

Not sure, can xarray break compatibility with dask <1.1.0 with some future version? Otherwise I guess there needs to be some legacy code in xarray which calls the right function.

Output of xr.show_versions()

This might help to avoid confusion. data_vars is always a mapping, not a mapping, a variable or a tuple.

Passing just a tuple, does not work of course. But for xarray newbies, this might be less obvious and the error message is also not easy to interpret: ```

xr.Dataset(('dim1', np.ones(5))) ... TypeError: unhashable type: 'numpy.ndarray' ```

The correct version of the example above should be: ```

xr.Dataset({'myvar': ('dim1', np.ones(5))})
<xarray.Dataset> Dimensions: (dim1: 5) Dimensions without coordinates: dim1 Data variables: myvar (dim1) float64 1.0 1.0 1.0 1.0 1.0 ```

See also pull-request #2860 - the same typo was at many places. Sorry, I have missed the other places when sending the first PR.

This is just a minor fix of a wrong error message. Please let me know if you think that this is worth testing in unit tests.

Before:

```

import xarray as xr d = xr.DataArray([1,2,3]) d.interp(1) ... ValueError: the first argument to .rename must be a dictionary ```

After: ```

import xarray as xr d = xr.DataArray([1,2,3]) d.interp(1) ...

ValueError: the first argument to .interp must be a dictionary ```