issue_comments

How to handle dimensions and coordinate names when assigning groups is clearly one of the important design decisions here. It's obvious that data variables should be grouped but less clear how to handle dimensions/coordinates.

I would be +1 for allowing tuples for data variables names but not for dimensions/coordinates names. It indeed looks like that using tuples for the latter would be a greater source of confusion and would add too much complexity for only little (or no real?) benefit.

I'd be fine with raising an error when loading a netCDF4 file which have groups with conflicting dimensions or when assigning an incompatible Dataset as a new group (e.g., ds['flux'] = incompatible_ds).

For groups that share common dimensions/coordinates with some differences, a data structure built on top of Dataset (like DatasetGroup or DatasetNode) would be more appropriate I think.

@darothen you might be interested by the discussion we had here, although it doesn't solve anything related to selection across similar Dataset objects.

I think that the collection of Dataset objects with like-dimensions that you suggest is indeed different than the tree-like structure within a dataset that is proposed here (the latter still using a unique set of dimensions and coordinates).

Both approaches may co-exist, though. I can imagine the case where we have (1) a set of, e.g., grid-search or monte-carlo model runs and (2) for each model run we have diagnostic variables defined in different places on the grid (e.g., nodes, edges...). The tuple-defined groups within a Dataset is useful for 2 and the collection of Dataset objects is useful for 1.

As pointed out by @shoyer, such a collection of Dataset objects might be (preferably) implemented outside of xarray.

Just want to say that I'm very enthusiastic about this!

Like @lamorton, I also find myself having a lot of variables with names containing the name(s) of their "group(s)".

My initial idea was also to keep flat datasets and add some logic to get/set groups, but it wasn't very clear and well explained.

One important reason to keep the tree-like structure within a dataset is that it provides some assurance to the recipient of the dataset that all the variables 'belong' in the same coordinate space.

Makes perfect sense!

I also find the idea of using tuples very clever! @shoyer do you have an idea on how it would work with serialization to netCDF? We would also have to decide how to display groups in the repr of the flat dataset...

@lamorton @shoyer unless you want to open a PR, I'd be willing to start working on this.

For example, how do groups get updated when you slice, aggregate or concatenate datasets?

Yep once again I haven't thought about all the implications this would have! This would indeed add much complexity at the end.

I'll try to follow you suggestion of building another data structure, for example - correct me if it's a wrong approach too - a DatasetGroup class which would be very similar to netCDF4.Group or h5py.Group but which would here contain a single Dataset.