issue_comments

Thanks for launching this discussion @TomNicholas ! I'm a core dev of pytroll/satpy which handles earth observing satellite data. I got interested in DataTree because we have data from the same instruments available at mulitple resolution, hence not fitting into a single Dataset. For use Option 1 is probably feeling better. Even when having data at multiple resolutions, it is still a limited number of resolutions and hence splitting them in groups is the natural way of going I would say. We do not use the features you mention in Zarr or GRIB, as a majority of the satellite data we use is provided in netcdf nowadays. Don't hesitate to ask if you want to know more or if something is unclear, we are really interested in these developments, so if we can help that way...

I am not sure I completely understand option 2, but option 1 seems a better fit to what we are doing at ArviZ (so far we are managing quite well with the InferenceData mentioned above which is a collection of independent xarray datasets). In our case, well defined selection for multiple variables at the same time (i.e. at the dataset level) is very useful.

I was also wondering what changes (if any) would each option imply when using apply_ufunc

For scientific imaging, i.e. biomicroscopy, medical imaging, where xarray compatibility is being considered in the NGFF, it would be helpful to avoid unnecessary divergence by ensuring the proposed hierarchical storage is compatible. This would mean:

1. Each scale / group can be independently treated as an xarray.Dataset.
2. They are organized in such a way that the collection of scales can be referenced as it is now, i.e. as a collection of paths,

{ “multiscales”: [ { “datasets” : [ {"path": "0"}, {"path": "1"}, {"path": "2"}, {"path": "3"}, {"path": "4"} ] “version” : “0.1” } ] }

Whoa, that sounds awesome! Thanks for the heads up :) Definitely could be quite handy, looking forward to seeing how this develops. @rocco8773 this should be interesting for you as well :)

Here are some biomedical papers that are using ArviZ and therefore xarray even if most don't cite xarray and some don't cite ArviZ either. Topics are quite disperse: covid, psychology, biomolecules, oncology...

I volunteer to contribute writing to this from the condensed matter / sychrotron user facility perspective.

Not really sure if there is anything we can do from ArviZ to help with that, if there is let us know and we'll do our best cc @percygautam

@jhamman @joshmoore a prototype to bring together XArray and OME-Zarr/NGFF with multiple groups: https://github.com/OpenImaging/miqa/blob/master/server/scripts/compress_encode.py

a. For example, our friends over at Arviz have a InferenceData structure composed of multiple Datasets that is represented on-disk using NetCDF groups: https://arviz-devs.github.io/arviz/notebooks/XarrayforArviZ.html

Just a note that this link has moved to: https://arviz-devs.github.io/arviz/getting_started/XarrayforArviZ.html