issue_comments

FWIW I think it would be better in xarray or a separate package, at least at the moment, just because I don't have a lot of time right now for OSS and need to keep Zarr as lean as possible.

On Thursday, February 23, 2017, Martin Durant notifications@github.com wrote:

@alimanfoo https://github.com/alimanfoo , do you think this work would make more sense as part of zarr rather than as part of xarray?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/1223#issuecomment-281990573, or mute the thread https://github.com/notifications/unsubscribe-auth/AAq8QoeCQOn7WvB8gtLP5Bs6cifIKRQiks5rfYjSgaJpZM4Lp0yH .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health http://cggh.org The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: alimanfoo@googlemail.com Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721

@alimanfoo , do you think this work would make more sense as part of zarr rather than as part of xarray?

True, xarray_to_zarr is unchanged from before. The dataset functions could supercede, since a single xarray is just a special case of a dataset; or we could decide that for the special case it is worth having short-cut functions. I was worried about the number of metadata files being created, since on a remote system like S3, there is a large overhead to reading many small files.

Yep, that looks good. I was wondering about the xarray_to_zarr() function?

On Wednesday, February 22, 2017, Martin Durant notifications@github.com wrote:

@alimanfoo https://github.com/alimanfoo , in the new dataset save function, I do exactly [as you suggest] (https://gist.github.com/ martindurant/06a1e98c91f0033c4649a48a2f943390#file-zarr_xarr-py-L168), with everything getting put as a dict into the main zarr group attributes, with special attribute names "attrs" for the data-set root, "coords" for the set of coordinate objects and "variables" for the set of variables objects (all of these have their own attributes in xarray).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/1223#issuecomment-281813651, or mute the thread https://github.com/notifications/unsubscribe-auth/AAq8QqSXNzQkrR0xOhhcp9QxWUIkz8Teks5rfKvggaJpZM4Lp0yH .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health http://cggh.org The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: alimanfoo@googlemail.com Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721

@alimanfoo , in the new dataset save function, I do exactly as you suggest, with everything getting put as a dict into the main zarr group attributes, with special attribute names "attrs" for the data-set root, "coords" for the set of coordinate objects and "variables" for the set of variables objects (all of these have their own attributes in xarray).

Just to say this is looking neat.

For storing an xarray.DataArray, do you think it would be possible to do away with pickling up all metadata and storing in the .xarray resource? Specifically I'm wondering if this could all be stored as attributes on the Zarr array, with some conventions for special xarray attribute names? I'm guessing there must be some conventions for storing all this metadata as attributes in an HDF5 (netCDF) file, it would potentially be nice to mirror that as much as possible?

On Sat, Feb 11, 2017 at 10:56 PM, Martin Durant notifications@github.com wrote:

I have developed my example a little to sidestep subclassing you suggest, which seemed tricky to implement.

Please see https://gist.github.com/martindurant/ 06a1e98c91f0033c4649a48a2f943390 (dataset_to/from_zarr functions)

I can use the zarr groups structure to mirror at least typical use of xarrays: variables, coordinates and sets of attributes on each. I have tested this with s3 too, stealing a little code from dask to show the idea.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/1223#issuecomment-279181938, or mute the thread https://github.com/notifications/unsubscribe-auth/AAq8QtydMLiMvgETYyaVF5D1CLb-4ot4ks5rbjy5gaJpZM4Lp0yH .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health http://cggh.org The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: alimanfoo@googlemail.com Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721

I have developed my example a little to sidestep subclassing you suggest, which seemed tricky to implement.

Please see https://gist.github.com/martindurant/06a1e98c91f0033c4649a48a2f943390 (dataset_to/from_zarr functions)

I can use the zarr groups structure to mirror at least typical use of xarrays: variables, coordinates and sets of attributes on each. I have tested this with s3 too, stealing a little code from dask to show the idea.

Happy to help if there's anything to do on the zarr side.

On Fri, 20 Jan 2017 at 23:47, Matthew Rocklin notifications@github.com wrote:

Also cc @alimanfoo https://github.com/alimanfoo

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/1223#issuecomment-274209930, or mute the thread https://github.com/notifications/unsubscribe-auth/AAq8QlwtJQ_OKOekveWuYtLmpR-caHvgks5rUUeTgaJpZM4Lp0yH .

3: a json-like representation such as used by the hidden .xarray item would also do.