issue_comments

To understand what's going on here, it may be helpful to look at what's going on inside dask: ``` In [16]: x = np.arange(5)

In [17]: da = xr.DataArray(np.ones(5), coords=[('x', x)]).chunk(-1)

In [18]: da Out[18]: <xarray.DataArray (x: 5)> dask.array<shape=(5,), dtype=float64, chunksize=(5,)> Coordinates: * x (x) int64 0 1 2 3 4

In [19]: da.reindex({'x': np.arange(20)}) Out[19]: <xarray.DataArray (x: 20)> dask.array<shape=(20,), dtype=float64, chunksize=(20,)> Coordinates: * x (x) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

In [20]: da.reindex({'x': np.arange(20)}).data.dask Out[20]: <dask.sharedict.ShareDict at 0x3201d72e8>

In [21]: dict(da.reindex({'x': np.arange(20)}).data.dask) Out[21]: {('where-8e0018fae0773d202c09fde132189347', 0): (subgraph_callable, ('eq-8167293bb8136be2934a8bf111095d8f', 0), array(nan), ('getitem-0eab360ba0dee5a5c3fbded0fdfd70e3', 0)), ('eq-8167293bb8136be2934a8bf111095d8f', 0): (subgraph_callable, ('array-5ddc8bae2e6cf87c0bac846c6da4d27f', 0), -1), ('array-5ddc8bae2e6cf87c0bac846c6da4d27f', 0): array([ 0, 1, 2, 3, 4, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]), ('xarray-<this-array>-765734894ab8f05a57335ea1064da549', 0): (<function dask.array.core.getter(a, b, asarray=True, lock=None)>, 'xarray-<this-array>-765734894ab8f05a57335ea1064da549', (slice(0, 5, None),)), 'xarray-<this-array>-765734894ab8f05a57335ea1064da549': ImplicitToExplicitIndexingAdapter(array=NumpyIndexingAdapter(array=array([1., 1., 1., 1., 1.]))), ('getitem-0eab360ba0dee5a5c3fbded0fdfd70e3', 0): (<function _operator.getitem(a, b, /)>, ('xarray-<this-array>-765734894ab8f05a57335ea1064da549', 0), (array([0, 1, 2, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4]),))} ```

Xarary isn't controlling chunk sizes directly, but it's turns reindex({'x': x2}) into an indexing operation like data[np.array([0, 1, 2, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4])] under the covers, with the last element repeated -- in your case 100000 times. See xarray.Variable._getitem_with_mask for the implementation on the xarray side.

The alternative design would be to append an array of all NaNs along one axis, but on average I think the current implementation is faster and results in more contiguous chunks -- it's quite common to intersperse missing indices with reindex() and alternating indexed/missing values can result in tiny chunks. Even then I think you would probably run into performance issues -- I don't think dask.array.full() uses a default chunk size.

We could also conceivably put some heuristics to control chunking for this in xarray, but I'd rather do it upstream in dask.array, if possible (xarray tries to avoid thinking about chunks).