html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue https://github.com/pydata/xarray/issues/1223#issuecomment-282031922,https://api.github.com/repos/pydata/xarray/issues/1223,282031922,MDEyOklzc3VlQ29tbWVudDI4MjAzMTkyMg==,703554,2017-02-23T15:55:38Z,2017-02-23T15:55:38Z,CONTRIBUTOR,"FWIW I think it would be better in xarray or a separate package, at least at the moment, just because I don't have a lot of time right now for OSS and need to keep Zarr as lean as possible. On Thursday, February 23, 2017, Martin Durant wrote: > @alimanfoo , do you think this work would > make more sense as part of zarr rather than as part of xarray? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > , or mute > the thread > > . > -- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: alimanfoo@googlemail.com Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,202260275 https://github.com/pydata/xarray/issues/1223#issuecomment-281990573,https://api.github.com/repos/pydata/xarray/issues/1223,281990573,MDEyOklzc3VlQ29tbWVudDI4MTk5MDU3Mw==,6042212,2017-02-23T13:25:36Z,2017-02-23T13:25:36Z,CONTRIBUTOR,"@alimanfoo , do you think this work would make more sense as part of zarr rather than as part of xarray?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,202260275 https://github.com/pydata/xarray/issues/1223#issuecomment-281860859,https://api.github.com/repos/pydata/xarray/issues/1223,281860859,MDEyOklzc3VlQ29tbWVudDI4MTg2MDg1OQ==,6042212,2017-02-23T01:25:52Z,2017-02-23T01:25:52Z,CONTRIBUTOR,"True, xarray_to_zarr is unchanged from before. The dataset functions could supercede, since a single xarray is just a special case of a dataset; or we could decide that for the special case it is worth having short-cut functions. I was worried about the number of metadata files being created, since on a remote system like S3, there is a large overhead to reading many small files.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,202260275 https://github.com/pydata/xarray/issues/1223#issuecomment-281829618,https://api.github.com/repos/pydata/xarray/issues/1223,281829618,MDEyOklzc3VlQ29tbWVudDI4MTgyOTYxOA==,703554,2017-02-22T22:43:52Z,2017-02-22T22:43:52Z,CONTRIBUTOR,"Yep, that looks good. I was wondering about the xarray_to_zarr() function? On Wednesday, February 22, 2017, Martin Durant wrote: > @alimanfoo , in the new dataset save > function, I do exactly [as you suggest] (https://gist.github.com/ > martindurant/06a1e98c91f0033c4649a48a2f943390#file-zarr_xarr-py-L168), > with everything getting put as a dict into the main zarr group attributes, > with special attribute names ""attrs"" for the data-set root, ""coords"" for > the set of coordinate objects and ""variables"" for the set of variables > objects (all of these have their own attributes in xarray). > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > , or mute > the thread > > . > -- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: alimanfoo@googlemail.com Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,202260275 https://github.com/pydata/xarray/issues/1223#issuecomment-281813651,https://api.github.com/repos/pydata/xarray/issues/1223,281813651,MDEyOklzc3VlQ29tbWVudDI4MTgxMzY1MQ==,6042212,2017-02-22T21:42:49Z,2017-02-22T21:43:05Z,CONTRIBUTOR,"@alimanfoo , in the new dataset save function, I do exactly [as you suggest](https://gist.github.com/martindurant/06a1e98c91f0033c4649a48a2f943390#file-zarr_xarr-py-L168), with everything getting put as a dict into the main zarr group attributes, with special attribute names ""attrs"" for the data-set root, ""coords"" for the set of coordinate objects and ""variables"" for the set of variables objects (all of these have their own attributes in xarray).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,202260275 https://github.com/pydata/xarray/issues/1223#issuecomment-281496902,https://api.github.com/repos/pydata/xarray/issues/1223,281496902,MDEyOklzc3VlQ29tbWVudDI4MTQ5NjkwMg==,703554,2017-02-21T22:05:39Z,2017-02-21T22:05:39Z,CONTRIBUTOR,"Just to say this is looking neat. For storing an xarray.DataArray, do you think it would be possible to do away with pickling up all metadata and storing in the .xarray resource? Specifically I'm wondering if this could all be stored as attributes on the Zarr array, with some conventions for special xarray attribute names? I'm guessing there must be some conventions for storing all this metadata as attributes in an HDF5 (netCDF) file, it would potentially be nice to mirror that as much as possible? On Sat, Feb 11, 2017 at 10:56 PM, Martin Durant wrote: > I have developed my example a little to sidestep subclassing you suggest, > which seemed tricky to implement. > > Please see https://gist.github.com/martindurant/ > 06a1e98c91f0033c4649a48a2f943390 > (dataset_to/from_zarr functions) > > I can use the zarr groups structure to mirror at least typical use of > xarrays: variables, coordinates and sets of attributes on each. I have > tested this with s3 too, stealing a little code from dask to show the idea. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > , or mute > the thread > > . > -- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: alimanfoo@googlemail.com Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,202260275 https://github.com/pydata/xarray/issues/1223#issuecomment-279181938,https://api.github.com/repos/pydata/xarray/issues/1223,279181938,MDEyOklzc3VlQ29tbWVudDI3OTE4MTkzOA==,6042212,2017-02-11T22:56:56Z,2017-02-11T22:56:56Z,CONTRIBUTOR,"I have developed my example a little to sidestep subclassing you suggest, which seemed tricky to implement. Please see https://gist.github.com/martindurant/06a1e98c91f0033c4649a48a2f943390 (dataset_to/from_zarr functions) I can use the zarr groups structure to mirror at least typical use of xarrays: variables, coordinates and sets of attributes on each. I have tested this with s3 too, stealing a little code from dask to show the idea. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,202260275 https://github.com/pydata/xarray/issues/1223#issuecomment-274230041,https://api.github.com/repos/pydata/xarray/issues/1223,274230041,MDEyOklzc3VlQ29tbWVudDI3NDIzMDA0MQ==,1217238,2017-01-21T03:18:38Z,2017-01-21T03:21:19Z,MEMBER,"@martindurant thanks for posting this as an issue -- I didn't get a notification from your ping in the gist. I agree that serializing xarray objects to zarr should be pretty straightforward and seems quite useful. To properly handle edge cases like strange data types (e.g., datetime64 or object) and `Dataset` objects, we probably want to integrate this with xarray existing conventions handling and DataStore interface. This will be good motivation for me to finish up my refactor in https://github.com/pydata/xarray/pull/1087 -- right now the interface is a bit more complex than needed, and doesn't do a good job of abstracting details like whether file formats need locking. So we could either directly [write a DataStore](https://github.com/pydata/xarray/blob/master/xarray/backends/) or write a separate ""znetcdf"" or ""netzdf"" module that implements an interface similar to [h5netcdf](https://github.com/shoyer/h5netcdf) (which itself is a thin wrapper on top of h5py). All things being equal, I would prefer the later approach, because people seem to find these intermediate interfaces useful, and it would help clarify the specification of the file format vs. details of how xarray uses it. As far as the spec goes, I agree that JSON is the sensible file format. Really, all we need on top of zarr is: - specified dimensions sizes, stored at the group level (`Dict[str, int]`) - a list of dimension names associated with each array (`List[str]`) - a small amount of validation logic to ensure that dimensions used on an array exist (on the array's group or one of its parents) and are consistent This could make sense either as part of zarr or a separate library. I would lean towards putting it in zarr only because that would be slightly more convenient, as we could safely make use of subclassing to add the extra functionality. zarr already handles hierarchies, arrays and metadata, which is most of the hard work. I'm certainly quite open to integrate experimental data formats like this one into xarray, but ultimately of course it depends on interest from the community. This wouldn't even necessarily need to live in xarray proper (though that would be fine, too). For example, @rabernat wrote a DataStore for loading MIT GCM outputs (https://github.com/xgcm/xmitgcm).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,202260275 https://github.com/pydata/xarray/issues/1223#issuecomment-274214755,https://api.github.com/repos/pydata/xarray/issues/1223,274214755,MDEyOklzc3VlQ29tbWVudDI3NDIxNDc1NQ==,703554,2017-01-21T00:24:27Z,2017-01-21T00:24:27Z,CONTRIBUTOR,"Happy to help if there's anything to do on the zarr side. On Fri, 20 Jan 2017 at 23:47, Matthew Rocklin wrote: > Also cc @alimanfoo > > > > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > , or mute > the thread > > . > > > > > > > > > > > > > > > > > > > ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,202260275 https://github.com/pydata/xarray/issues/1223#issuecomment-274209930,https://api.github.com/repos/pydata/xarray/issues/1223,274209930,MDEyOklzc3VlQ29tbWVudDI3NDIwOTkzMA==,306380,2017-01-20T23:47:29Z,2017-01-20T23:47:29Z,MEMBER,Also cc @alimanfoo ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,202260275 https://github.com/pydata/xarray/issues/1223#issuecomment-274202189,https://api.github.com/repos/pydata/xarray/issues/1223,274202189,MDEyOklzc3VlQ29tbWVudDI3NDIwMjE4OQ==,6042212,2017-01-20T22:57:07Z,2017-01-20T22:57:07Z,CONTRIBUTOR,3: a json-like representation such as used by the hidden .xarray item would also do.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,202260275 https://github.com/pydata/xarray/issues/1223#issuecomment-274200419,https://api.github.com/repos/pydata/xarray/issues/1223,274200419,MDEyOklzc3VlQ29tbWVudDI3NDIwMDQxOQ==,306380,2017-01-20T22:46:44Z,2017-01-20T22:46:44Z,MEMBER,"This looks pretty cool to me. I expected it to be harder to encode xarray into zarr. Some thoughts/comments: 1. *Is* it harder to encode a full xarray into zarr? Are there cases that are not covered by this example that are likely to occur in the wild (mostly a question for @shoyer) 2. I guess one major thing missing is storing full Dataset objects rather than just DataArrays. I suspect that scientific users want to keep all of the variables and coordinates in a single artifact 2. It would be nice to avoid using pickle if possible, so that the data could be cross-language. 3. How open is the XArray community to adding experimental to/from_zarr methods? 4. Eventually we probably want to do `lazy_value = da.store(..., compute=False)` and then compute all of them at once @pwolfram @rabernat @jhamman ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,202260275