issue_comments

No, that should be updated. Thanks for pointing it!

Thanks for taking a look. I'm writing some docs on reshaping today, then will merge this and issue the new release / rename if I have time.

On Sat, Jan 16, 2016 at 11:23 PM, Joe Hamman notifications@github.com wrote:

@shoyer - this is cool. I just breezed through the code and didn't see anything that jumped out at me. The main API comment I have is about the repr which has already been discussed and identified in #719.

Reply to this email directly or view it on GitHub: https://github.com/pydata/xarray/pull/702#issuecomment-172298171

@shoyer - this is cool. I just breezed through the code and didn't see anything that jumped out at me. The main API comment I have is about the repr which has already been discussed and identified in #719.

makes sense about dask.array.dropna

though I think you should dropna if at all possible (or have an option at least)

it IS a bit suprising to get back the full index not sure how common that will be in practice esp if u r stacking multiple levels

finally - think about only supporting sequential stacking as it conceptually makes more sense

hmm, is dask.array dropna not implemented? I don't see why it couldn't conceptually be done (though a bit unfamiliar with the impl)

We have a dropna in xarray. The problem is that for dask arrays, you need to know the shape of the result. With dropna, you don't know the shape until you've actually done the computation, so it can't be done lazily.

hmm, is dask.array dropna not implemented? I don't see why it couldn't conceptually be done (though a bit unfamiliar with the impl) - set_index takes 'data' and makes it an 'index', so that is orthogonal. It would make a new Coordinate. reset_index would do the converse. - stack/unstack effectively take existing Coordinates and transform between them.

ok makes sense.

@jreback thanks for the comments!

I think the repr, though technically accurate, is a bit misleading. lists of tuples is really only useful as a MI, so why not actually indicate that

Agreed -- this is part of my "better repr" TODO.

stack/unstack (as in [9]) is not idempotent, as you are reconstituting the full cartesian product of levels. This seems a bit odd though (pandas can do this because its is separately tracking what is actually in the index, via the labels), I don't think you have this though?

This is true, and definitely worth noting as a compatibility break. But I do think we have a good reason for this: pandas's stack uses dropna (effectively) to drop unused levels, but this operation cannot be done lazily with dask.array. I am happy to force users to do a non-lazy dropna explicitly.

these ops are really analogs of set_index/reset_index, rather than stack/unstack, so might be a bit confusing (though I think I get why you are doing it this way), it makes more sense esp for multi-dim. Maybe explain this in the pandas guide?

I'm not quite sure what you mean here -- set_index/reset_index seem independent of these to me (though they would definitely also be worth adding!). The difference I see: - set_index: make 1d variables part of a (multi)-index along their existing axis - stack: combine orthogonal indexes (along different axes) into a multi-index

couple of comments: - I think the repr, though technically accurate, is a bit misleading. lists of tuples is really only useful as a MI, so why not actually indicate that - stack/unstack (as in [9]) is not idempotent, as you are reconstituting the full cartesian product of levels. This seems a bit odd though (pandas can do this because its is separately tracking what is actually in the index, via the labels), I don't think you have this though? - these ops are really analogs of set_index/reset_index, rather than stack/unstack, so might be a bit confusing (though I think I get why you are doing it this way), it makes more sense esp for multi-dim. Maybe explain this in the pandas guide?

I think the api is great. Stack / unstack is a nice way to describe the operation of aggregating coordinates.

Any opinions, even on the API here? I'd like to merge this this week...

Big :+1: from me. This seems like a feature with enormous potential.

It would be nice to eventually build a full example showing how stack can be combined with lazy loading / dask to do out-of-core PCA on a large geophysical dataset (e.g., identify El Nino)

I have an example notebook for doing svd on a sea-surface-temperature field which should be pretty easy to adapt to these new methods. (Currently I just switch over to numpy for the actual svd.)

I'd like to this into the next release in something close to its current state. It's not as full featured as I would eventually like (see checklist above), but it's enough to be useful, and I'd like to get v0.7 (with the new name) out next week.