issue_comments

Indeed, it looks like this works now. Extending the example from the first post: In [3]: ds.chunk({'x': 5}).thedata.groupby('thegroup').mean() Out[3]: <xarray.DataArray 'thedata' (thegroup: 2)> dask.array<shape=(2,), dtype=float64, chunksize=(1,)> Coordinates: * thegroup (thegroup) object False True

This is what I was looking for:

Frozen(SortedKeysDict({'allpoints': (1, 1, 1, 1, 1......(allpoints)....., 1, 1), 'T': (11L,)}))

So in this case (where the chunk size is already 1), dask.array.reshape could actually work fine and the error is unnecessary (we don't have the exploding task issue). So this could potentially be fixed upstream in dask.

For now, the best work-around (because you don't have any memory concerns) is to "rechunk" into a single block along the last axis before reshaping, e.g., .chunk(allpoints=259200) or .chunk(allpoints=1e9) (or something arbitrarily large).

@byersiiasa What matters for dask's reshape is the array shape and chunk shape, all of which you should see when you print a dask.array (or xarray.DataArray containing one). What is the size of the chunking along time and allpoints?

That array is loaded in numpy already - can you share the dask version? On Mon, Mar 13, 2017 at 2:57 AM byersiiasa notifications@github.com wrote:

<xarray.DataArray 'dis' (time: 30, allpoints: 259200)> array([[ 9.969210e+36, 9.969210e+36, 9.969210e+36, ..., 9.969210e+36, 9.969210e+36, 9.969210e+36], [ 9.969210e+36, 9.969210e+36, 9.969210e+36, ..., 9.969210e+36, 9.969210e+36, 9.969210e+36], [ 9.969210e+36, 9.969210e+36, 9.969210e+36, ..., 9.969210e+36, 9.969210e+36, 9.969210e+36], ..., [ 9.969210e+36, 9.969210e+36, 9.969210e+36, ..., 9.969210e+36, 9.969210e+36, 9.969210e+36], [ 9.969210e+36, 9.969210e+36, 9.969210e+36, ..., 9.969210e+36, 9.969210e+36, 9.969210e+36], [ 9.969210e+36, 9.969210e+36, 9.969210e+36, ..., 9.969210e+36, 9.969210e+36, 9.969210e+36]]) Coordinates: * time (time) datetime64[ns] 1971-01-01 1972-01-01 1973-01-01 ... * allpoints (allpoints) MultiIndex - lon (allpoints) float64 -179.8 -179.8 -179.8 -179.8 -179.8 -179.8 ... - lat (allpoints) float64 89.75 89.25 88.75 88.25 87.75 87.25 86.75 ...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/1026#issuecomment-286062113, or mute the thread https://github.com/notifications/unsubscribe-auth/ABKS1pd_uTiUQLRXjBhR7D06uvkkKJBDks5rlRLxgaJpZM4KMB0C .

@byersiiasa can you share what stacked.dis looks like?

It would look something like this: 1. Verify that chunks are the same on all dask arrays to be stacked. 2. Use np.ravel with map_blocks to flatten each block independently. 3. Construct the appropriate (non-sorted) MultiIndex to label the flattened elements.

This was an intentional change -- see https://github.com/dask/dask/pull/1469

Previously, we created lots of teeny tasks, which tended to negate any out of core benefits. The problem is that reshape promises an order to the elements it reshape which tends to split across existing chunks of dask arrays.

We could work around this in xarray by adding custom logic to stack for keeping chunks together when reshaping, but we can't do this upstream in dask because we need to make sure we keep all the arrays aligned.