issue_comments

@jhamman There's already #1551 open but I need to heavily rethink it to cater for all the various use cases offered by the data_vars and coords parameters of concat().

Back to banging my head on it. Expect a heavy rewrite of combine.py. Can't say an ETA but it's going to be a fair amount of hours.

P.S. need to put #1522 as a prerequisite in order not to lose my sanity, as this change is very much hitting on the same nails

Just realised that you can concat(data_vars='different') and have the exact same problem on data_vars :|

Also, with "different" I realised that you're comparing the variable contents twice, once in _calc_concat_over and another time at the end of _dataset_concat. This also slows down concat on pure-numpy backend.

Need more time to work on this... I'll be on holiday for the next week with no access to my PC; should be able to continue after the 12 Sept.

Getting closer. The problem is in xarray.concat, which resolves non-index dask coords, TWICE, even if it should not resolve them at all (as alignment should be done on index coords only?)

``` import xarray import numpy import dask.array

def kernel(label): print("Kernel [%s] invoked!" % label) return numpy.array([1, 2])

a = dask.array.Array(name='a', dask={('a', 0): (kernel, 'a')}, chunks=((2, ), ), dtype=int) b = dask.array.Array(name='b', dask={('b', 0): (kernel, 'b')}, chunks=((2, ), ), dtype=int)

ds0 = xarray.Dataset(coords={'x': ('x', [1, 2]), 'y': ('x', a)}) ds1 = xarray.Dataset(coords={'x': ('x', [1, 2]), 'y': ('x', b)}) xarray.concat([ds0, ds1], dim='z') Output: Kernel [a] invoked! Kernel [b] invoked! Kernel [b] invoked!Kernel [a] invoked!

<xarray.Dataset> Dimensions: (x: 2) Coordinates: * x (x) int64 1 2 y (x) int64 dask.array<shape=(2,), chunksize=(2,)> Data variables: empty ```

As suspected, the problem is caused specifically by non-index coords:

``` import xarray import numpy

data = numpy.random.randint(1<<63, size=1000000)

for r in range(50): ds = xarray.Dataset( coords={'r': [r], 'c': data, 'otherindex': data}, data_vars={'data': (('r', 'c'), data.reshape(1, data.size))}) ds.to_netcdf('fast.%02d.nc' % r) del ds['otherindex'] ds.coords['nonindex'] = ('c', data) ds.to_netcdf('slow.%02d.nc' % r)

def load_coords(ds): for coord in ds.coords.values(): coord.load() return ds %time xarray.open_mfdataset('fast..nc') %time xarray.open_mfdataset('fast..nc', preprocess=load_coords) %time xarray.open_mfdataset('slow..nc') %time xarray.open_mfdataset('slow..nc', preprocess=load_coords) output: CPU times: user 332 ms, sys: 88 ms, total: 420 ms Wall time: 420 ms CPU times: user 348 ms, sys: 84 ms, total: 432 ms Wall time: 430 ms CPU times: user 1.13 s, sys: 200 ms, total: 1.33 s Wall time: 1.07 s CPU times: user 596 ms, sys: 104 ms, total: 700 ms Wall time: 697 ms ```

change open_dataset() to always eagerly load the coords to memory, regardless of the chunks parameter. Is there any valid use case where lazy coords are actually desirable?

This also leads to another inefficiency of open_dataset(chunks=...), where you may have your data e.g. shape=(50000, 2**30), chunks=(1, 2**30). If you pass the chunks above to open_dataset, it will break down the coords on the first dim into dask arrays of 1 element - which hardly benefits anybody. Things get worse if the dataset is compressed with zlib or whatever, but only the data vars were chunked at the moment of writing. Am I correct in understanding that the whole coord var will be read from disk 50000 times over?