issue_comments

We should clarify that the aim of Index objects is to make more efficient all the operations made in the (discrete or continuous) space defined by the coordinate labels. That space is distinct from the discrete space defined by array element locations.

I think this should be one of the first things said. It defines what all the following discussion of Indexes does and does not affect.

I've tried to explain it in the "Index base class" section and the sections below, but maybe it should be emphasized more?

Yeah I think you do actually have that one covered, I just included it as another example of a naive question that everyone will have that is worth heading off very explicitly.

When I load the "air" tutorial data and it shows a Float64Index and DateTime64Index, where did they come from?

I guess you mean it is shown through ds.indexes?

ds.xindexes (vs. ds.indexes) still needs to be added in the docs (in a later PR?), which hopefully will address your concern here.

I meant like when did these indexes get automatically built? (Presumably on coordinate assignment)

Maybe Index also deserves its own entry there, where we could explain what indexes are, how they are different from variables (coordinates), how they are used or accessed in Xarray, etc.

1000% yes we need a page that explains what Index objects are, what they do, and how they work, and how they are handled automatically by default. This is pre-requisite knowledge (which apparently I don't have :sweat_smile: ) before trying to build your own custom index.

Overall, I think that the whole "Xarray Internals" section could be streamlined beyond a bunch of loosely-coupled document pages.

Probably, but having a loosely coupled page for each aspect of the internals would be a good initial aim.

I agree that we need more examples, but I also think that too much examples may tend to make things more confused.

That's why I like the "Explanation" vs "How-to" vs "Tutorials" distinction: use minimal code in the "Explanation" section (this PR) but put multiple more complex examples under "How to create a functionally-derived index", "how-to create a lazy index" etc.

Is it possible to do that with Sphinx / RST?

No idea, but that does look cool!

pydata/xarray your feedback would be very much appreciated! I've been into this for quite some time, so there may be things that seem obvious to me but that you can still find very confusing or non-intuitive. It would then deserve some extra or better explanation.

I find the way in which the Index objects are involved in a method call (e.g. sel) pretty opaque still. (Bear in mind I don't think I've ever contributed a PR to xarray that touched indexes.py or indexing.py :sweat_smile:)

• When is my CustomIndex object consulted? Is it potentially for all basic operations (concat, join, align, indexing, etc?) If I passed some kind of NotImplementedIndex which only defined .from_variables, what functionality would be left in xarray?
• Why does there not need to be an index (in .indexes) if I do indexing with e.g. .isel but have no coordinates?
• If I index along two dimensions simultaneously (.isel(x=1, y=2)) does that correspond to two separate Index consultations?
• How is an xarray.PandasIndex different from a pd.Index?
• When I load the "air" tutorial data and it shows a Float64Index and DateTime64Index, where did they come from?
• Finally it's not great that to explain Index objects we have to assume the user knows what xarray.Variable is, but Variable is still not really public API, and certainly isn't documented as comprehensively as DataArray and Dataset are.

Perhaps these questions are too specific to xarray's internals but I do think there should be some kind of mental model given as to the role the Index objects play. (This could be a white lie, similar to how our page on data structures says that Dataset objects contain DataArray objects when actually they technically don't.)

I also think we need multiple simple examples. How about - HelloWorldIndex, that just prints "I'm calling HelloWorldIndex.isel !" etc. - PeriodicBoundaryIndex (a partial/simpler implementation of #7031), - A simple functionally-derived index, that consults a dynamically-called exponential function or something. - Some example of a custom multi-index, perhaps some kind of 2D lat-lon thing? Or how about something that represents 2D image distortion, like using it creates a fisheye effect?

I find it helps to think of documentation using this 4-part system. This PR should cover "Explanation" pretty well, but we should still aim for other content to better cover "Tutorial", "How-to Guides", and "Reference". "Tutorial" could be like a notebook walking through creating a simple index (e.g. PeriodicIndex) from scratch, explaining and fixing errors as they arise (like @dcherian did for apply_ufunc). "How-to Guides" might be specific to other more advanced custom index examples. I guess "Reference" here is just having really clear docstrings on the possible methods of Index somewhere (we can't really do that with an ABC though can we?).

That all said, this is already a great start!