html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/4118#issuecomment-904817641,https://api.github.com/repos/pydata/xarray/issues/4118,904817641,IC_kwDOAMm_X8417mvp,35968931,2021-08-24T17:00:24Z,2022-05-19T16:33:26Z,MEMBER,"So I had a crack at making a full `DataTree` class - you can find it in [this repo](https://github.com/TomNicholas/datatree).
It's based on @benbovy's [`DatasetNode` example](https://gist.github.com/benbovy/92e7c76220af1aaa4b3a0b65374e233a) - the basic idea is that each tree node wraps a single `Dataset`. The differences are that this effort:
- [Uses a NodeMixin from anytree](https://github.com/TomNicholas/datatree/issues/7) for the tree structure,
- Implements path-like and tag-like getting and setting,
- Has functions for mapping user-supplied functions over every node in the tree,
- Automatically dispatches `xarray.Dataset`'s API over every node in the tree (such as `.isel` or `__add__`),
- Has a bunch of tests,
- Has a printable representation that currently looks like this:
![]()
Some limitations of the approach I used are:
- Each dataset in the tree is entirely separate, so doing something like `dt.sel(time=50)` would require each Dataset in that subtree to have it's own coordinate called `'time'`. (That's normally useful though because then
`'time'` can be a different resolution on each ds),
- While you can access nodes via tags, the underlying implementation is in terms of paths, so `('folder1', 'folder2')` points to a different node than `('folder2', 'folder1')`,
- There's no support for [symbolic nodes yet](https://github.com/TomNicholas/datatree/issues/5), and I'm unsure if this design can allow for loops or not.
You can create a `DataTree` object in 3 ways:
1) Load from a netCDF file that has groups via `open_datatree()`,
2) Using the init method of `DataTree`, which accepts a nested dictionary of Datasets,
3) Manually create individual nodes with `DataNode()` and specify their relationships to each other, either by setting `.parent` and `.children` attributes, or through `__get/setitem__` access, e.g.
`dt['path/to/node'] = DataNode('node_name', data=xr.Dataset())`.
It's about 70% working, but some things I could do with some help with are:
1) ~Fundamental [design questions](https://github.com/TomNicholas/datatree/issues?q=is%3Aissue+is%3Aopen+label%3A%22design+question%22) about the class structure, such as whether `DataTree` [should be a subclass](https://github.com/TomNicholas/datatree/issues/2) of `Dataset`?~
2) ~Getting [arithmetic](https://github.com/TomNicholas/datatree/pull/24) and [ufuncs](https://github.com/TomNicholas/datatree/issues/25) to act properly on the whole tree~,
3) ~[Saving a tree to a single netCDF file](https://github.com/TomNicholas/datatree/issues/16)~, (thanks Joe!)
4) ~Setting up CI and all that jazz~, (thanks Joe again!)
5) ~Setting up basic docs.~
There will definitely be many bugs, but any thoughts or input appreciated!
","{""total_count"": 8, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 8, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-1047944213,https://api.github.com/repos/pydata/xarray/issues/4118,1047944213,IC_kwDOAMm_X84-dlwV,35968931,2022-02-22T15:58:48Z,2022-02-22T15:58:48Z,MEMBER,"Also thanks @OriolAbril , it's useful to have an ArViz perspective.
> I was also wondering what changes (if any) would each option imply when using `apply_ufunc`
I see `apply_ufunc` as a `Variable`-level operation - i.e. it doesn't know about the relationship between different Variables unless you explicit feed it multiple variables. So therefore whether we choose model 1 or 2 probably doesn't affect `apply_ufunc` much.
In either case I imagine all we might need to do is slightly extend `apply_ufunc` to also map over variables in a group of a tree if given one, and provide examples of using [`map_over_subtree`](https://github.com/TomNicholas/datatree/blob/3beff56f653ba430b41b0aab971571072ff30334/datatree/mapping.py#L106) or similar to map your `apply_ufunc` operation over multiple groups in a tree. If the user is trying to do something more complicated (like getting one variable from one level of a tree and another variable from another level, then feeding both into `apply_ufunc`) then I would just make the user responsible for fetching the variables in that case, and also for putting the results back into the intended place in the tree.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-1047932340,https://api.github.com/repos/pydata/xarray/issues/4118,1047932340,IC_kwDOAMm_X84-di20,35968931,2022-02-22T15:47:15Z,2022-02-22T15:50:41Z,MEMBER,"Hi @LunarLanding , thanks for your ideas!
> For this, it would make more sense to be able to have dimensions ( with optional labels and coordinates ) assigned to nodes (and these would be inherited by any descendants).
It sounds a bit like what you are suggesting is essentially a model in which dimensions are explicit objects, which can be referred to from other groups, like in netCDF. (NetCDF has ""dimension IDs"".)
This would be a bit of a departure from the model that `xarray.Dataset` currently uses, because right now dimensions aren't really unique entities, they are just a collective label for a shared dimension of a set of `Variable` objects.
> Often I run a function over a dataset, with each call outputing a hierarchical data structure, containing fixed dimensions in the best cases and variable length in the worst.
By ""variable"" length, do you mean that the length of dimensions differs between variables in the same group, or just that you don't know the length of the dimension in advance?
Is there a specific use case which you think would require explicit dimensions to solve?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-1043638105,https://api.github.com/repos/pydata/xarray/issues/4118,1043638105,IC_kwDOAMm_X84-NKdZ,35968931,2022-02-17T23:47:44Z,2022-02-17T23:47:44Z,MEMBER,"> This is only true for flat netCDF files, once you introduce groups in a netCDF AND accept CF conventions the DataGroup approach can map 100% of the files, while the DataTree approach fails on a (admittedly small) class of them.
@alexamici can you expand on the role of the CF conventions in this statement? Are you talking about CF conventions allowing one variable in one group to refer to dimension present in another group, or something else?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-1042769595,https://api.github.com/repos/pydata/xarray/issues/4118,1042769595,IC_kwDOAMm_X84-J2a7,5821660,2022-02-17T09:58:18Z,2022-02-17T09:58:18Z,MEMBER,"> in the representation I use the fully qualified name for the dimension / coordinate, but the corresponding `DataArray` will use the basename, e.g. both array will have `lat` as a coordinate. Sorry for te confusion, I need to add more context to the README.
Thanks for clarifying. I'm wondering if that can be a source of misunderstanding. How should the user differentiate that? I mean finally those dimensions which have the same name `lat` are different entities and it should be possible to tell them apart somehow. I think I'm slowly getting to the bottom of this (representation in dictionaries, duplicate keys) and I really need to look into the implementation. I'll open an issue over at xarray-datagroup if I have more questions to not clutter the discussion here.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-1042753800,https://api.github.com/repos/pydata/xarray/issues/4118,1042753800,IC_kwDOAMm_X84-JykI,226037,2022-02-17T09:41:29Z,2022-02-17T09:53:55Z,MEMBER,"@kmuehlbauer in the representation I use the fully qualified name for the dimension / coordinate, but the corresponding `DataArray` will use the basename, e.g. both array will have `lat` as a coordinate. Sorry for the confusion, I need to add more context to the README.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-1042731962,https://api.github.com/repos/pydata/xarray/issues/4118,1042731962,IC_kwDOAMm_X84-JtO6,5821660,2022-02-17T09:17:55Z,2022-02-17T09:17:55Z,MEMBER,"@alexamici
> * in [netCDF following the CF conventions for groups](https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#groups), it is legal for an array to refer to a dimension or a coordinate in a different group and so arrays in the same group may have dimensions with the same name, but different size / coordinate values, (this was the orginal motivation to explore the DataGroup approach)
I'm having difficulties to understand your above point wrt to the scoping rules from the above CF document. Shouldn't it be impossible to create two arrays (in the same group) having dimensions with exactly the same name from different groups? Looking at the example here https://github.com/alexamici/xarray-datagroup there are coordinates with name ""/lat"" vs ""lat"". Aren't that two different names? Maybe I'm missing something essential here.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-1042656377,https://api.github.com/repos/pydata/xarray/issues/4118,1042656377,IC_kwDOAMm_X84-Jax5,226037,2022-02-17T07:39:15Z,2022-02-17T08:17:51Z,MEMBER,"@TomNicholas (cc @mraspaud)
> Do you have use cases which one of these designs could handle but the other couldn't?
The two main classes of on-disk formats that, I know of, which cannot be always represented in the ""group is a Dataset"" approach are:
- in [netCDF following the CF conventions for groups](https://cfconventions.org/Data/cf-conventions/cf-conventions-1.9/cf-conventions.html#groups), it is legal for an array to refer to a dimension or a coordinate in a different group and so arrays in the same group may have dimensions with the same name, but different size / coordinate values, (this was the orginal motivation to explore the DataGroup approach)
- the current spec for the [Next-generation file formats (NGFF)](https://ngff.openmicroscopy.org) for bio-imaging has all scales of the same 5D data in the same group. (cc @joshmoore)
I don't have an example at hand, but my impression is that satellite products that use HDF5 file format also place arrays with inconsistent dimensions / coordinates in the same group.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-1042664227,https://api.github.com/repos/pydata/xarray/issues/4118,1042664227,IC_kwDOAMm_X84-Jcsj,226037,2022-02-17T07:52:17Z,2022-02-17T07:53:13Z,MEMBER,"@TomNicholas I also have a few comments on the comparison:
> * **Option (1) - Each group is a Dataset**
>
> * Model maps more directly onto netCDF (though still not exactly, because netCDF has dimensions as separate objects)
This is only true for flat netCDF files, once you introduce groups in a netCDF AND accept CF conventions the DataGroup approach can map 100% of the files, while the DataTree approach fails on a (admittedly small) class of them.
> * Enforcing consistency between variables guarantees certain operations are always well-defined (in particular selection via an integer index like in `.isel`).
> * Guarantees that all valid operations on a Dataset are also valid operations on a single group of a DataTree - so API can be essentially identical to Dataset.
Both points are only true for the DataArray in a single group, once you broadcast any operation to subgroups the two implementations would share the same limitations (dimensions in subgroups can be inconsistent in both cases).
In my opinion the advantage for the DataTree is minimal.
> * Metadata (i.e. `.attrs`) are arguably most useful when set at this level
The two approach are identical in this respect, group attributes are mapped in the same way to DataTree and DataGroup
I share your views on all other points.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-1042660100,https://api.github.com/repos/pydata/xarray/issues/4118,1042660100,IC_kwDOAMm_X84-JbsE,1217238,2022-02-17T07:45:24Z,2022-02-17T07:45:24Z,MEMBER,"One thing that came up in our discussion about this in the developer
meeting today is that we could also pretty easily expose a ""low level"" API
for IO using dictionaries of xarray.Variable objects. This intermediate
representation could be useful for cleaning up data into a form suitable
for conversion into Dataset objects.
On Wed, Feb 16, 2022 at 11:39 PM Alessandro Amici ***@***.***>
wrote:
> @TomNicholas (cc @mraspaud
> )
>
> Do you have use cases which one of these designs could handle but the
> other couldn't?
>
> The two main classes of on-disk formats that, I know of, which cannot be
> always represented in the ""group is a Dataset"" approach are:
>
> - in netCDF following the CF conventions for groups
> ,
> it is legal for an array to refer to a dimension or a coordinate in a
> different group and so arrays in the same group may have dimensions with
> the same name, but different size / coordinate values,
> - the current spec for the Next-generation file formats (NGFF)
> for bio-imaging has all scales of
> the same 5D data in the same group.
>
> I don't have an example at hand, but my impression is that satellite
> products that use HDF5 file format also place arrays with inconsistent
> dimensions / coordinates in the same group.
>
> —
> Reply to this email directly, view it on GitHub
> ,
> or unsubscribe
>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-1039572760,https://api.github.com/repos/pydata/xarray/issues/4118,1039572760,IC_kwDOAMm_X8499p8Y,35968931,2022-02-14T21:19:56Z,2022-02-14T21:40:21Z,MEMBER,"We would like some opinions from the community on two different possible models for a tree-like structure in xarray.
A tree contains many groups, but the question is what constraints should be imposed on the contents of those groups.
- **Option (1) - Each group is a Dataset**
- Means that within each group the same restrictions apply as currently do within a single dataset, i.e. each dimension name is only associated with a single length, so there is effectively a common set of dimensions which variables can depend on.
- Can't represent all files, in particular can't represent a filetype where groups are allowed to have variables with inconsistent length dimensions (e.g. [Zarr stores allow this](https://github.com/ome/ngff/issues/48#issuecomment-1031416259) as all arrays are independent.)
- Model maps more directly onto netCDF (though still not exactly, because netCDF has dimensions as separate objects)
- This means that sometimes you might need to put variables in ajdacent groups in the same level of the tree, when you might rather want them together in the same group.
- Enforcing consistency between variables guarantees certain operations are always well-defined (in particular selection via an integer index like in `.isel`).
- Guarantees that all valid operations on a Dataset are also valid operations on a single group of a DataTree - so API can be essentially identical to Dataset.
- Metadata (i.e. `.attrs`) are arguably most useful when set at this level
- Mental model is a (nested) dict of Datasets
- Prototype is [DataTree](https://github.com/TomNicholas/datatree)
- **Option (2) - Variables within groups are unconstrained**
- Means that within a single group each Variable can have any dimensions, of any length. There is no requirement that two variables which both depend on a dimension called ""x"" have to have the same length, one variable can have `.sizes['x']=10` and the other have `.sizes['x']=20`.
- The main advantage of this is that it can represent a wider set of files (including all Zarr stores and a wider set of GRIB files)
- Model maps more directly onto HDF5
- Doesn't enforce the (arguably fairly arbitrary) constraint that if variables have a dimension of the same name, that dimension must also be the same length
- Without consistency selection becomes ill-defined, but many other operations are fine (e.g. taking `.mean()`)
- Mental model is a (nested) dict of dicts of DataArrays
- Prototype is [xarray-DataGroups](https://github.com/alexamici/xarray-datagroup)
This is by no means the only question, and we have various choices to make within these options.
The questions for the potential users here are:
- Do you have use cases which one of these designs could handle but the other couldn't?
- How important to you is being able to support all valid files of these certain formats?
- Which of these designs is clearer/more intuitive/more appealing to you?
(@alexamici , @shoyer, @jhamman, @aurghs please edit this comment to add anything I've missed)
","{""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 2, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-905472692,https://api.github.com/repos/pydata/xarray/issues/4118,905472692,IC_kwDOAMm_X841-Gq0,35968931,2021-08-25T12:50:04Z,2021-08-25T13:02:10Z,MEMBER,"Thanks @benbovy !
> For rich/html reprs, I think that we could take much inspiration from some of the dask reprs shown in this blog post.
I don't know much about HTML, but graphs where you can mouseover nodes to see node information sound awesome!
> what is the rationale of having two separate classes DataTree and DataNode?
They aren't separate: `DataNode` is merely a (perhaps badly-named) [pointer](https://github.com/TomNicholas/datatree/blob/a11ad48c723271fbb55d1e9ca1c4afd49d6d3b93/datatree/datatree.py#L744) to a [second init method](https://github.com/TomNicholas/datatree/blob/a11ad48c723271fbb55d1e9ca1c4afd49d6d3b93/datatree/datatree.py#L402) for the same `DataTree` class.
The idea was that creating a single node of a tree by specifying only its `(name, dataset, parent, children)` attributes was conceptually different to ""I have loads of datasets, and I want to arrange them all into one big tree using path-like addresses"", so I made two different init methods on `DataTree` to cover that. The idea was from the `xarray.Dataset._construct_direct()` classmethod, which creates a new instance of a Dataset by directly setting attributes like `(variables, coord_names, dims, attrs)`. That is an internal classmethod though, and isn't externally exposed like `DataNode()`.
We could just merge the two signatures into one `__init__` method though, or use a less confusing name (I just didn't want `DataTree._init_single_node(name, data, parent)` everywhere in my tests.) Also internally it's nice to have a separate `._init_single_node()` method because that's (a) closer to the `super().__init__()` defined by `TreeNode`, and (b) doesn't require calling the fairly complex getting and setting methods.
> Could those classes be merged somehow?
They were originally separate (I had `DataTree` and `DatasetNode`, where the former was a subclass of the latter), but then I merged them together in [Condense DatasetNode and DataTree into a single DataTree class #11](https://github.com/TomNicholas/datatree/pull/11).
> Zarr (abstract data store) has no such separate class and uses a regular zarr.hierarchy.Group as the root.
Good to know that other nested structures took a similar approach. I think that as we want to be able to save and load any subtree even after changing parents etc. then we ideally don't want to treat any one node as special.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-905427176,https://api.github.com/repos/pydata/xarray/issues/4118,905427176,IC_kwDOAMm_X84197jo,4160723,2021-08-25T11:47:10Z,2021-08-25T11:47:10Z,MEMBER,"Great work @TomNicholas!
For rich/html reprs, I think that we could take much inspiration from some of the dask reprs shown in this [blog post](https://blog.dask.org/2021/08/23/gsoc-2021-project).
I haven't looked at your repository in detail yet, but I have one general question about the design: what is the rationale of having two separate classes `DataTree` and `DataNode`? Could those classes be merged somehow? To me it seems that any node could be the root of a (sub)tree. `h5py` has a separate `File` object that [does double duty as the HDF5 root group, and serves as entry point into the file](https://docs.h5py.org/en/stable/high/group.html), but in the case of Xarray maybe this is less relevant since Xarray object are abstract data containers? Zarr (abstract data store) has no such separate class and uses a regular [`zarr.hierarchy.Group` as the root](https://zarr.readthedocs.io/en/stable/tutorial.html#groups). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-904987705,https://api.github.com/repos/pydata/xarray/issues/4118,904987705,IC_kwDOAMm_X8418QQ5,35968931,2021-08-24T21:25:17Z,2021-08-24T21:25:37Z,MEMBER,"Thanks @jhamman - expect things to break as I keep realizing certain methods have to be defined differently from in Dataset for things to work.
Help with 3 would be especially appreciated, as at the moment whilst I can open and alter a file with groups, I can't save my resulting tree :sweat_smile: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-904970588,https://api.github.com/repos/pydata/xarray/issues/4118,904970588,IC_kwDOAMm_X8418MFc,2443309,2021-08-24T21:00:33Z,2021-08-24T21:00:33Z,MEMBER,Thanks @TomNicholas! I've just been starting to look into this. I'm going to give it a spin and would be happy to help with your numbers 3 and 4. ,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-901954045,https://api.github.com/repos/pydata/xarray/issues/4118,901954045,IC_kwDOAMm_X841wrn9,35968931,2021-08-19T14:16:45Z,2021-08-19T14:16:45Z,MEMBER,"Oh excellent, thanks for the clarification Stephan!
On Thu, 19 Aug 2021, 00:23 Stephan Hoyer, ***@***.***> wrote:
> However, if one of the variables has the same name as one of the groups
> (which I think is permitted in the netCDF format), then there is no easy
> way to access all the elements whilst retaining the nice syntax.
>
> NetCDF does *not* allow variables and groups with the same name, e..g,
>
> import netCDF4
> nc = netCDF4.Dataset('testing.nc', 'w')nc.createVariable('foo', float)nc.createGroup('foo')# RuntimeError: NetCDF: String match to name in use
>
> I'm pretty sure this is also prohibited for all HDF5 files, just like how
> you can't have a directory and file with the same name on most filesystems.
>
> —
> You are receiving this because you are on a team that was mentioned.
> Reply to this email directly, view it on GitHub
> , or
> unsubscribe
>
> .
>
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-901598698,https://api.github.com/repos/pydata/xarray/issues/4118,901598698,IC_kwDOAMm_X841vU3q,1217238,2021-08-19T04:23:15Z,2021-08-19T04:23:15Z,MEMBER,"> However, if one of the variables has the same name as one of the groups (which I think is permitted in the netCDF format), then there is no easy way to access all the elements whilst retaining the nice syntax.
NetCDF does *not* allow variables and groups with the same name, e..g,
```python
import netCDF4
nc = netCDF4.Dataset('testing.nc', 'w')
nc.createVariable('foo', float)
nc.createGroup('foo')
# RuntimeError: NetCDF: String match to name in use
```
I'm pretty sure this is also prohibited for all HDF5 files, just like how you can't have a directory and file with the same name on most filesystems.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-901594249,https://api.github.com/repos/pydata/xarray/issues/4118,901594249,IC_kwDOAMm_X841vTyJ,35968931,2021-08-19T04:10:30Z,2021-08-19T04:10:30Z,MEMBER,"I think that xarray's current use of both dict-like access and attribute-like access for variables makes representing a general netCDF file in a single `DataTree` incompatible with the nice syntax that @emilbiju originally suggested.
Consider a tree with a node structure for a hypothetical `DataTree` object `dt` that looks something like
```python
DataTree(""root"")
|-- DatasetNode(""weather"")
| |-- DatasetNode(""temperature"")
| | |-- DataArrayNode(""sea_surface_temperature"")
| | |-- DataArrayNode(""dew_point_temperature"")
| |-- DataArrayNode(""wind_speed"")
|-- DataArrayNode(""population"")
```
We ideally want to be able to seamlessly access both subtrees and individual variables via chains of keys, e.g.
`weather_subtree = dt['weather']`, and `wind_speed_da = dt['weather']['wind_speed']`. (We want that so that each subtree behaves as much like an `xarray.Dataset` as possible, with respect to mapping functions over all its child nodes and so on.)
This particular example is fine, and would correspond to a netCDF file with groups ""root"", ""root/weather"", and ""root/weather/temperature"", plus the four stored DataArray variables.
However, if one of the variables has the same name as one of the groups (which I think is permitted in the netCDF format), then there is no easy way to access all the elements whilst retaining the nice syntax. For example consider
```python
DataTree(""root"")
|-- DatasetNode(""A"")
| |-- DatasetNode(""B"")
| | |-- DataArrayNode(""foo"")
| | |-- DataArrayNode(""bar"")
| |-- DataArrayNode(""B"")
|-- DataArrayNode(""C"")
```
Now we have a key collision between the group named ""B"" and the DataArray named ""B"", i.e. `dt['A']['B']` is ambiguous.
We can't just forbid this type of tree because then there would be netCDF files that we couldn't represent as a `DataTree`, so we would not have the property `netCDF -> xarray.DataTree -> netCDF` in general.
We can't use different types of access (e.g. `subtree = dt.A.B` for the subtree and `da = dt.A['B']` for the variable, because we've already given up the `.B` namespace to also point to the variable (i.e. same location as `['B']`). If we break that convention it's going to be very confusing for users who are expecting the root of the `DataTree` to behave like `xarray.Dataset` currently does.
(We could divide access through `__call__` like `ds['A']('B')` but [that wouldn't be very pythonic](https://github.com/pydata/xarray/issues/3939)).
The only way I can see around this is to hide a node's data variables behind a `.ds` property (i.e. `da = dt['A'].ds['B']`), or get groups via a dedicated method (i.e. `subtree = dt.get_child('A')`), but those are so much more ugly and less intuitive that it feels like a shame to have to do that.
It sounds like @emilbiju avoided this by not satisfying `netCDF -> xarray.DataTree -> netCDF`:
> (Instead of using netCDF4 groups for encoding the Datatree ... within the netCDF file, it would exist just as a Dataset)
so I'm wondering if anyone else has other suggestions or thoughts?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-873492892,https://api.github.com/repos/pydata/xarray/issues/4118,873492892,MDEyOklzc3VlQ29tbWVudDg3MzQ5Mjg5Mg==,35968931,2021-07-04T00:51:19Z,2021-07-04T00:51:19Z,MEMBER,"Some other thoughts about tags:
1) Does the definition of tags include variable names of DataArrays? I think it should.
2) As @martinitus mentioned, a `DataTree` containing only leaves with only 1 tag each is effectively a Dataset. I wonder if Dataset could be refactored to be a special case of a more general DataTree, possibly as a subclass?
3) Selecting via tags would need to allow a distinction between ""get me all leaves with these exact tags"" and ""get me all leaves whose tags include these ones"". Maybe `dt.choose_only(tags)` and `dt.choose_all(tags)`?
4) The latter type of tag-based access would make plotting different leaves against one another easier too - given a multi-resolution (or multi-model) datatree like this:
```
dt
|-- high_res
| |-- temperature
| |-- CO2
|-- medium_res
| |-- temperature
| |-- CO2
|-- low_res
| |-- temperature
| |-- CO2
```
then assuming that the definition of tags included the DataArray variable names, then
`dt.choose_all('temperature').plot.line(x='time')`
would select all leaves with a tempature tag, check that the temperature DataArrays had the same dimensions (but no need for any `time` coordinates to share size or values), and then plot them against one another on the same axes. This would be so useful - I would say this use case is 90% of the reason users iterate over dictionaries of datasets currently.
5) With a tag-based system you can create cycles of tags, like A&B, B&C, C&A, which you can't really do with hierarchical trees. I don't think that actually causes any problems though...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-873316602,https://api.github.com/repos/pydata/xarray/issues/4118,873316602,MDEyOklzc3VlQ29tbWVudDg3MzMxNjYwMg==,1217238,2021-07-03T00:40:55Z,2021-07-03T00:40:55Z,MEMBER,"> if you used tags wouldn't you lose the ability to round-trip a netCDF file with groups?
That sounds right to me -- a downside of tags is that they can't be (uniquely) expressed in a hierarchical arrangement like those found in HDF5/netCDF4 files.
But if this is a better way to organize data in memory, we could consider how to make an adapter layer for on disk storage.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-873307873,https://api.github.com/repos/pydata/xarray/issues/4118,873307873,MDEyOklzc3VlQ29tbWVudDg3MzMwNzg3Mw==,35968931,2021-07-02T23:54:09Z,2021-07-02T23:54:09Z,MEMBER,"@shoyer if you used tags wouldn't you lose the ability to round-trip a netCDF file with groups? When you read in the groups from the file you would be throwing information away by going from a hierarchy A/B to simply tags A&B, and there wouldn't be a way to restore that before calling `.to_netcdf()` would there?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-873231425,https://api.github.com/repos/pydata/xarray/issues/4118,873231425,MDEyOklzc3VlQ29tbWVudDg3MzIzMTQyNQ==,35968931,2021-07-02T20:05:06Z,2021-07-02T20:05:06Z,MEMBER,"> I think using tags is a really interesting alternative to hierarchies. I don't have a clear sense of the overall tradeoffs, though.
That is interesting. I think there is an argument for using a hierarchical model to map onto the full netCDF data model with groups, but perhaps methods to select elements via tags could be included too, for the best of both?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-873227326,https://api.github.com/repos/pydata/xarray/issues/4118,873227326,MDEyOklzc3VlQ29tbWVudDg3MzIyNzMyNg==,1217238,2021-07-02T19:55:31Z,2021-07-02T19:55:31Z,MEMBER,"@martinitus raises a really interesting point about tags vs hierarchical structures over in https://github.com/pydata/xarray/issues/1092#issuecomment-868324949
> However, one point I didn't see in the discussion is the following:
>
> Hierarchical structures often force a user to come up with some arbitrary order of hierarchy levels. The classical example is document filing: do you put your health insurance documents under /insurance/health/2021, 2021/health/insurance,....?
>
> One solution to that is a tagging of documents instead of putting them into a hierarchy. This would give the full flexibility to retrieve any flat DataSet out of a TaggedDataSet by specifying the set of tags that the individual DataArrays must be listed under.
I think using tags is a really interesting alternative to hierarchies. I don't have a clear sense of the overall tradeoffs, though.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-873179375,https://api.github.com/repos/pydata/xarray/issues/4118,873179375,MDEyOklzc3VlQ29tbWVudDg3MzE3OTM3NQ==,35968931,2021-07-02T18:22:49Z,2021-07-02T18:22:49Z,MEMBER,"Flagging another possible use case, this time in Magnetic Confinement Fusion: representing the IMAS data model.
IMAS is currently closed-source (being part of the ITER project), but there is a big push to make it open-source and the standard data model for tokamak plasma data.
I'm not very familiar with IMAS (@smithsp and @orso82 are more so), but it is hierarchical. There is some more information in appendix A3 of [this paper](https://iopscience.iop.org/article/10.1088/1741-4326/abb918/meta), which talks about ""taking advantage of the homogeneity of grid sizes that is commonly found across arrays of structures"", which sounds very closely related to the `DataTree` proposal.
This might allow the `xarray.DataTree` to do more of the heavy-lifting within [OMAS](https://github.com/gafusion/omas) (which already uses xarray, and is intended to be compatible with IMAS).
","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-846137752,https://api.github.com/repos/pydata/xarray/issues/4118,846137752,MDEyOklzc3VlQ29tbWVudDg0NjEzNzc1Mg==,2448579,2021-05-21T17:58:38Z,2021-05-21T17:58:38Z,MEMBER,cc @d-v-b and https://github.com/JaneliaSciComp/xarray-multiscale,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-808366093,https://api.github.com/repos/pydata/xarray/issues/4118,808366093,MDEyOklzc3VlQ29tbWVudDgwODM2NjA5Mw==,35968931,2021-03-26T16:47:53Z,2021-03-26T16:47:53Z,MEMBER,"This sounds like an interesting project - I'm also about to be able to work on xarray much more directly (thanks @rabernat ).
Should I add this as another xarray project board alongside explicit indexes and so on?
I wonder if this could find another domain use case in plasmapy as part of the overall `plasma` object @StanczakDominik? At the very least this would allow you to store all the various equilibrium and diagnostics information that goes in an EFIT file.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-807908489,https://api.github.com/repos/pydata/xarray/issues/4118,807908489,MDEyOklzc3VlQ29tbWVudDgwNzkwODQ4OQ==,1217238,2021-03-26T03:24:48Z,2021-03-26T03:24:48Z,MEMBER,"I'm excited to see this coming together! I would be happy to advise as well...
Side note: at some point, this would probably be worth adding to Xarray's official roadmap.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-806954634,https://api.github.com/repos/pydata/xarray/issues/4118,806954634,MDEyOklzc3VlQ29tbWVudDgwNjk1NDYzNA==,2448579,2021-03-25T15:30:53Z,2021-03-25T15:30:53Z,MEMBER,I can shoulder part of the load and help is definitely needed. LOI is due on Tuesday. I'll take a stab this evening and post a link.,"{""total_count"": 4, ""+1"": 4, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-806701802,https://api.github.com/repos/pydata/xarray/issues/4118,806701802,MDEyOklzc3VlQ29tbWVudDgwNjcwMTgwMg==,1197350,2021-03-25T13:01:56Z,2021-03-25T13:05:03Z,MEMBER,"So we have:
- Numerous promising prototypes to draw from
- A technical team who can write the proposal and execute the proposed work (@aurghs & @alexamici of B-open)
- Numerous supporting use cases from the bioimaging (@joshmoore), condensed matter (@tacaswell), and bayesian modeling (ArviZ; @OriolAbril) domains
We are just missing a PI, someone who is willing to put their name on top of the proposal and click submit. I have [gone on record](https://rabernat.medium.com/advising-and-collaborating-during-a-pandemic-and-sabbatical-ca9531b82b6d) as committed to not leading any new proposals this year. And in any case, this is a good opportunity for someone else from the @pydata/xarray core dev team to try on a leadership role.
","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-801785278,https://api.github.com/repos/pydata/xarray/issues/4118,801785278,MDEyOklzc3VlQ29tbWVudDgwMTc4NTI3OA==,4160723,2021-03-18T09:54:42Z,2021-03-18T09:54:42Z,MEMBER,"FWIW, a while ago I wrote a mock-up (and probably outdated) `DatasetNode` class:
https://gist.github.com/benbovy/92e7c76220af1aaa4b3a0b65374e233a ([nbviewer link](https://nbviewer.jupyter.org/gist/benbovy/92e7c76220af1aaa4b3a0b65374e233a))","{""total_count"": 3, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 3, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-801257666,https://api.github.com/repos/pydata/xarray/issues/4118,801257666,MDEyOklzc3VlQ29tbWVudDgwMTI1NzY2Ng==,2448579,2021-03-17T17:10:41Z,2021-03-17T17:10:41Z,MEMBER,"> But did we decide who might serve as PI for such a proposal?
No.
@emilbiju are you interested in open-sourcing your work?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-801240559,https://api.github.com/repos/pydata/xarray/issues/4118,801240559,MDEyOklzc3VlQ29tbWVudDgwMTI0MDU1OQ==,1197350,2021-03-17T16:47:20Z,2021-03-17T16:47:20Z,MEMBER,"On today's Xarray dev call, we discussed pursuing another CZI grant to support this feature in Xarray. The image pyramid use case would provide a strong link to the bioimaging community. @alexamici and the B-open folks seem enthusiastic.
I had to leave the meeting early, so I didn't hear the end of the conversation. But did we decide who might serve as PI for such a proposal?","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-755465523,https://api.github.com/repos/pydata/xarray/issues/4118,755465523,MDEyOklzc3VlQ29tbWVudDc1NTQ2NTUyMw==,2443309,2021-01-06T18:08:19Z,2021-01-06T18:08:19Z,MEMBER,"@joshmoore - based on https://github.com/pangeo-forge/pangeo-forge/pull/27#issuecomment-755397835, you may be interested in this issue. One way to do multiscale datasets in Xarray would be to use hierarchical groups (one group per scale). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-638481215,https://api.github.com/repos/pydata/xarray/issues/4118,638481215,MDEyOklzc3VlQ29tbWVudDYzODQ4MTIxNQ==,1217238,2020-06-03T21:52:53Z,2020-06-03T23:08:47Z,MEMBER,"The data model you sketch out here looks very similar to what we discussed in #1092. I agree that the semantics are well defined.
The main question in my mind is whether it would make more sense to make an entirely new data structure (e.g., `xarray.TreeDataset`) or add in a new feature like `groups` to the existing `xarray.Dataset`.
Probably a new data structure would be easier at this point, because would keep `Dataset` simpler and wouldn't break existing code that works on `xarray.Dataset`.","{""total_count"": 5, ""+1"": 5, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-638478790,https://api.github.com/repos/pydata/xarray/issues/4118,638478790,MDEyOklzc3VlQ29tbWVudDYzODQ3ODc5MA==,1217238,2020-06-03T21:46:48Z,2020-06-03T21:46:48Z,MEMBER,"I would be open to exploring adding a hierarchical data structure into xarray (on an experimental basis, to start), but it would need someone with serious interest and time to make it happen. Certainly there are plenty of use cases across various fields.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-637660689,https://api.github.com/repos/pydata/xarray/issues/4118,637660689,MDEyOklzc3VlQ29tbWVudDYzNzY2MDY4OQ==,2448579,2020-06-02T16:20:09Z,2020-06-02T16:20:09Z,MEMBER,"Thanks for writing this up @emilbiju . These are very interesting ideas
1. The nice thing about using NetCDF groups (or HDF5?) is that it is a standard and your data files are readable using other software.
2. So far, xarray has been reluctant to add ""groups"" or this kind of hierarchical organization because of all the additional complexity involved (#1092)
3. That said, there is definitely interest in a package that provides a high-level object composed of multiple xarray datasets (again #1092). So I encourage you to post your code online so others can try it out and iterate.
a. For example, our friends over at Arviz have a `InferenceData` structure composed of multiple Datasets that is represented on-disk using NetCDF groups: https://arviz-devs.github.io/arviz/notebooks/XarrayforArviZ.html
","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058
https://github.com/pydata/xarray/issues/4118#issuecomment-637163506,https://api.github.com/repos/pydata/xarray/issues/4118,637163506,MDEyOklzc3VlQ29tbWVudDYzNzE2MzUwNg==,2443309,2020-06-01T22:42:10Z,2020-06-01T22:42:10Z,MEMBER,@emilbiju - thanks for opening an issue here. You may want to take a look at the conversation in #1092. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,628719058