issue_comments

wait for zarr v2.13 release,

Done. And should be out on conda-forge later today.

Definitely interested and interestedly watching 🍿

jpivarski commented on Aug 10 It shouldn't be a subclass because it doesn't satisfy a substitution principle: ak.combinations(array: ak.Array, n: int) -> ak.Array, but ak.combinations(array: RaggedArray, n: int) -> ⊥ (at best, would raise an exception because RaggedArray isn't closed under ak.combinations).

A question from the perspective of Zarr (v3), does it make sense to think of this potential RaggedArray as a base extension that AwkwardArray could then build on top of? (i.e. the reverse) Or more something to keep separate and it's just a matter of the same library could be used to read (de/serialize) either?

More generally, :heart: for all of this, and interested to see how I/we can possibly help.

rabernat commented 7 hours ago My opinion is that we should not try to support the nczarr conventions directly. Xarray already supports nczarr via netCDF4. If netCDF4 can open the Zarr store, then Xarray can read it. ... I would turn this question around and ask: if netCDF4 supports access to these datasets directly, what's the advantage of xarray bypassing netCDF4 and opening them directly?

@malmans2 can chime in with his experience, but it seems that from the user point-of-view, not needing to know if something is an xarray-zarr or a nczarr would be kinder of us. Plus as said below, I do think it puts us on the path to defining a common spec.

Supporting nczarr directly would require lots of custom logic within xarray.

Mea culpa. I wasn't clear enough about the intent from my side at least, namely to support loading ARRAY_DIMENSIONS (or some other necessary subset) from nczarr rather than its entirety.

DennisHeimbigner commented 4 hours ago this is because xarray cannot handle subgroups.

I'll add as a side that work on the subgroups (i.e. datatree) is progressing in case any consideration needs to be included now rather than later.

Thanks, @max-sixty! Guess it just doesn't complete for those outside the org.

Thanks for the details, @DennisHeimbigner. But my reading of what you outline is that for some nczarr datasets, xarray will be able to open them. Correct? If so, there were always likely to be follow-on's to this issue when/if we identify critical edge cases. Perhaps for the moment, though, we can focus here on what we want to enable and what can be done straight-forwardly.

That likely makes this more a question for @shoyer, @jhamman, @rabernat et al. (sorry, no way to @-mention all the current devs) @malmans2 is probably in a good place to start updating the existing Zarr backend to also check for the nczarr files, but if there are strong opinions against or alternatives that would be preferred, it would be good to hear about them.

A heads up that https://github.com/zarr-developers/zarr-python/pull/755 reverses the default behavior of FSStore and so as of Zarr 2.8.3 this might "just work"?

I don't think I am familiar enough to really judge between the suggestions, @benbovy, but I'm intrigued. I think there's certainly something to be won just by having a data structure which says these arrays/datasets represent a multiscale series. One real benefit though will be when access of that structure can simplify the client code needed to interactively load that data, e.g. with prefetching.

Picking up on @dcherian's https://github.com/pydata/xarray/issues/4118#issuecomment-806954634 and @rabernat's https://github.com/ome/ngff/issues/48#issuecomment-833456889, Zarr was also accepted to the second round and certainly references this issue in case we want to sync up. (Apologies if I missed where that discussion moved.)

Happy to provide assistance on the image pyramid (i.e. "multiscale") use case.

Thanks for the link, @jhamman. The most immediate issue I ran into when trying to use xarray with OME-Zarr data does seem similar. A rough representation of one multiscale image is:

image_pyramid: |_ zyx_array_high_res |_ zyx_array_mid_res |_ zyx_array_low_res

but of course the x, y and z dimensions are of different sizes in each volume.

@rabernat : I don't assume you've found a workaround for this?