html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/pull/4035#issuecomment-721534803,https://api.github.com/repos/pydata/xarray/issues/4035,721534803,MDEyOklzc3VlQ29tbWVudDcyMTUzNDgwMw==,1217238,2020-11-04T06:18:35Z,2020-11-04T06:18:35Z,MEMBER,"> Should this checking be performed for all variables, or only for data_variables?
I agree that this requirement is a little surprising. The error is because otherwise you might be surprised that the array values for ""latitude"" and ""longtitude"" get overriden, rather than being checked for consistency. At least if you have to explicitly drop these variables (with the suggested call to `.drop()`) it is clear that they will neither be checked nor overriden in the Zarr store.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,613012939
https://github.com/pydata/xarray/pull/4035#issuecomment-719081563,https://api.github.com/repos/pydata/xarray/issues/4035,719081563,MDEyOklzc3VlQ29tbWVudDcxOTA4MTU2Mw==,1217238,2020-10-29T23:30:48Z,2020-10-29T23:30:48Z,MEMBER,"If there are no additional reviews or objections, I will merge this tomorrow.","{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,613012939
https://github.com/pydata/xarray/pull/4035#issuecomment-716041586,https://api.github.com/repos/pydata/xarray/issues/4035,716041586,MDEyOklzc3VlQ29tbWVudDcxNjA0MTU4Ng==,1217238,2020-10-24T19:12:33Z,2020-10-24T19:12:33Z,MEMBER,Anyone else want to take a look at this?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,613012939
https://github.com/pydata/xarray/pull/4035#issuecomment-712649636,https://api.github.com/repos/pydata/xarray/issues/4035,712649636,MDEyOklzc3VlQ29tbWVudDcxMjY0OTYzNg==,1217238,2020-10-20T07:21:29Z,2020-10-20T07:21:29Z,MEMBER,"OK, I think this is ready for a final review.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,613012939
https://github.com/pydata/xarray/pull/4035#issuecomment-711502852,https://api.github.com/repos/pydata/xarray/issues/4035,711502852,MDEyOklzc3VlQ29tbWVudDcxMTUwMjg1Mg==,1217238,2020-10-19T04:01:54Z,2020-10-19T04:01:54Z,MEMBER,"But yes, we've also been successfully using this for parallel writes for a few months now (aside from the race condition).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,613012939
https://github.com/pydata/xarray/pull/4035#issuecomment-711501546,https://api.github.com/repos/pydata/xarray/issues/4035,711501546,MDEyOklzc3VlQ29tbWVudDcxMTUwMTU0Ng==,1217238,2020-10-19T04:00:48Z,2020-10-19T04:00:48Z,MEMBER,"I just fixed a race condition with writing attributes. Let me spend a little bit of time responding to Ryan's review, and then I think we can submit it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,613012939
https://github.com/pydata/xarray/pull/4035#issuecomment-656637518,https://api.github.com/repos/pydata/xarray/issues/4035,656637518,MDEyOklzc3VlQ29tbWVudDY1NjYzNzUxOA==,1197350,2020-07-10T11:57:40Z,2020-07-10T11:57:40Z,MEMBER,"Zac, you may be interested in this thread
https://discourse.pangeo.io/t/best-practices-to-go-from-1000s-of-netcdf-files-to-analyses-on-a-hpc-cluster/588/32
Tom White managed to integrate dask with pywren via dask executor. This allows you to read / write zarr with lambda.
Sent from my iPhone
> On Jul 9, 2020, at 6:41 PM, Stephan Hoyer wrote:
>
>
> This looks nice. Is there a thought if this would work with functions as a service (GCP cloud functions, AWS Lambda, etc) for supporting parallel transformation from netcdf to zarr?
>
> I haven't used function as a service before, but yes, I imagine this might be useful for that sort of thing. As long as you can figure out the structure of the overall Zarr datasets ahead of time, you could use region to fill out different parts entirely independently.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,613012939
https://github.com/pydata/xarray/pull/4035#issuecomment-656385475,https://api.github.com/repos/pydata/xarray/issues/4035,656385475,MDEyOklzc3VlQ29tbWVudDY1NjM4NTQ3NQ==,1217238,2020-07-09T22:40:54Z,2020-07-09T22:43:34Z,MEMBER,"> This looks nice. Is there a thought if this would work with functions as a service (GCP cloud functions, AWS Lambda, etc) for supporting parallel transformation from netcdf to zarr?
I haven't used functions as a service before, but yes, I imagine this might be useful for that sort of thing. As long as you can figure out the structure of the overall Zarr datasets ahead of time, you could use `region` to fill out different parts entirely independently.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,613012939
https://github.com/pydata/xarray/pull/4035#issuecomment-646216297,https://api.github.com/repos/pydata/xarray/issues/4035,646216297,MDEyOklzc3VlQ29tbWVudDY0NjIxNjI5Nw==,1217238,2020-06-18T17:53:03Z,2020-06-18T17:53:03Z,MEMBER,"I've add error checking, tests and documentation, so this is ready for review now!
Take a look here for a rendered version of the new docs section:
https://xray--4035.org.readthedocs.build/en/4035/io.html#appending-to-existing-zarr-stores","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,613012939
https://github.com/pydata/xarray/pull/4035#issuecomment-627318136,https://api.github.com/repos/pydata/xarray/issues/4035,627318136,MDEyOklzc3VlQ29tbWVudDYyNzMxODEzNg==,1197350,2020-05-12T12:42:12Z,2020-05-12T12:42:37Z,MEMBER,"> A similar neat feature would be to read xarray datasets from regions of zarr groups w/o dask arrays.
@nbren12 - this has always been supported. Just call `open_zarr(..., chunks=False)` and then subset using `sel` / `isel`. ","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",,613012939
https://github.com/pydata/xarray/pull/4035#issuecomment-626207771,https://api.github.com/repos/pydata/xarray/issues/4035,626207771,MDEyOklzc3VlQ29tbWVudDYyNjIwNzc3MQ==,1217238,2020-05-09T17:14:46Z,2020-05-09T17:14:46Z,MEMBER,"> I'm curious how this interacts with dimension coordinates. Your example bypasses this. But what if dimension coordinates are present. How do we handle alignment issues? For example, what if I call `ds.to_zarr(path , region=selection)`, but the dimension coordinates of `ds` don't align with the dimension coordinates of the store at `path`""
It’s entirely unsafe. Currently the coordinates would be overridden with the new values , which is consistent with how to_netcdf() with mode=‘a’ works.
This is probably another good reason for requiring users to explicitly drop variables that don’t include a dimension in the selected region, because at least in that case there can be no user expectations about alignment with coordinates that don’t exist.
In the long term, it might make sense to make both to_netcdf and to_zarr check coordinates by alignment by default, but we wouldn’t want that in all cases, because sometimes users really do want to update variables.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,613012939
https://github.com/pydata/xarray/pull/4035#issuecomment-625865523,https://api.github.com/repos/pydata/xarray/issues/4035,625865523,MDEyOklzc3VlQ29tbWVudDYyNTg2NTUyMw==,1197350,2020-05-08T15:16:54Z,2020-05-08T15:16:54Z,MEMBER,"Stephan, this seems like a great addition. Thanks for getting it started!
I'm curious how this interacts with dimension coordinates. Your example bypasses this. But what if dimension coordinates are present. How do we handle alignment issues? For example, what if I call `ds.to_zarr(path , region=selection)`, but the dimension coordinates of `ds` don't align with the dimension coordinates of the store at `path`""
> 1. Officially document that the `compute` argument only controls writing array values, not metadata (at least for zarr).
:+1:
> 4\. Like (2), but raise an error instead of a warning. Require the user to explicitly drop them with `.drop()`. This is probably the safest behavior.
:+1:
I think only advanced users will want to use this feature.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,613012939