issue_comments

Ah, good call. The transpose currently in xarray would still be a problem though.

For the long term, I also understand that there isn't really a good way to check equality of two dask arrays. I wonder if dask's graph optimization could be used to "simplify" two dask arrays' graph separately and check the graph equality. For example, two dask arrays created by doing da.zeros((10, 10), chunks=2) + 5 should be theoretically equal because their dask graphs are made up of the same tasks.

Dask actually already does this canonicalization. If two arrays have the same name, they use the same dask graph, e.g., ``` In [5]: x = da.zeros((10, 10), chunks=2) + 5

In [6]: y = da.zeros((10, 10), chunks=2) + 5

In [7]: x.name Out[7]: 'add-f7441a0f46f5cf40458391cd08406c23'

In [8]: y.name Out[8]: 'add-f7441a0f46f5cf40458391cd08406c23' ```

So xarray could safely look at .name on dask arrays (e.g., inside Variable.equals or duck_array_ops.array_equiv) for determining that two dask arrays are the same, rather than merely using is to check if they are the same objects.

@shoyer Understood. That explains why something like this wasn't caught before, but what would be the best solution for a short term fix?

For the long term, I also understand that there isn't really a good way to check equality of two dask arrays. I wonder if dask's graph optimization could be used to "simplify" two dask arrays' graph separately and check the graph equality. For example, two dask arrays created by doing da.zeros((10, 10), chunks=2) + 5 should be theoretically equal because their dask graphs are made up of the same tasks.

Edit: "short term fix": What is the best way to avoid the unnecessary transpose? Or is this not even the right way to approach this? Change dask to avoid the unnecessary transpose or change xarray to not do the tranpose or something else?

The source of the problem here is that when combining objects, xarray needs to decide what coordinates should remain. Our current heuristic, which pre-dates dask support, was really designed for array in memory: we keep around coordinates if they are equal on both arguments, and remove them otherwise. In some cases we can avoid the computation, if we know that the coordinates are the same object.

I am open to ideas on how to make this work better.

FYI: @djhoese, you can inline code snippits using the permanent link to the source: https://github.com/pydata/xarray/blob/e5bb647637063153a7feb750793d6fd8fb58dda8/xarray/core/variable.py#L1223

Modifying this line to be:

python if dims == expanded_vars.sizes: return expanded_vars return expanded_var.transpose(*dims)

Then this issue is avoided for at least the + case.

Ok another update. In the previous example I accidentally added the lons coordinate DataArray with the dimensions redefined (('y', 'x'), lons2) which is technically redundant but it worked (no progress bar).

However, if I fix this redundancy and do:

python a = xr.DataArray(da.zeros((10, 10), chunks=2), dims=('y', 'x'), coords={'lons': lons2}) b = xr.DataArray(da.zeros((10, 10), chunks=2), dims=('y', 'x'), coords={'lons': lons2}) with ProgressBar(): c = a + b

I do get a progress bar again (lons2 is being computed). I've tracked it down to this transpose which is transposing when it doesn't need to which is causing the dask array to change:

https://github.com/pydata/xarray/blob/master/xarray/core/variable.py#L1223

I'm not sure if this would be considered a bug in dask or xarray. Also, not sure why the redundant version of the example worked.

Ok I'm getting a little more of an understanding on this. The main issue is that the dask array is not literally considered the same object because I'm creating the object twice. If I create a single dask array and pass it:

python lons = da.zeros((10, 10), chunks=2) a = xr.DataArray(da.zeros((10, 10), chunks=2), dims=('y', 'x'), coords={'y': np.arange(10), 'x': np.arange(10), 'lons': (('y', 'x'), lons)}) b = xr.DataArray(da.zeros((10, 10), chunks=2), dims=('y', 'x'), coords={'y': np.arange(10), 'x': np.arange(10), 'lons': (('y', 'x'), lons)})

I still get the progress bar because xarray is creating two new DataArray objects for this lons coordinate. So lons_data_arr.variable._data is not lons_data_arr2.variable._data causing the equivalency check here to fail.

If I make a single DataArray that becomes the coordinate variable then it seems to work:

python lons2 = xr.DataArray(lons, dims=('y', 'x')) a = xr.DataArray(da.zeros((10, 10), chunks=2), dims=('y', 'x'), coords={'y': np.arange(10), 'x': np.arange(10), 'lons': (('y', 'x'), lons2)}) b = xr.DataArray(da.zeros((10, 10), chunks=2), dims=('y', 'x'), coords={'y': np.arange(10), 'x': np.arange(10), 'lons': (('y', 'x'), lons2)})

I get no progress bar. ```