issue_comments: 656637518

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/pull/4035#issuecomment-656637518 https://api.github.com/repos/pydata/xarray/issues/4035 656637518 MDEyOklzc3VlQ29tbWVudDY1NjYzNzUxOA== 1197350 2020-07-10T11:57:40Z 2020-07-10T11:57:40Z MEMBER

Zac, you may be interested in this thread

https://discourse.pangeo.io/t/best-practices-to-go-from-1000s-of-netcdf-files-to-analyses-on-a-hpc-cluster/588/32

Tom White managed to integrate dask with pywren via dask executor. This allows you to read / write zarr with lambda.

Sent from my iPhone

On Jul 9, 2020, at 6:41 PM, Stephan Hoyer notifications@github.com wrote:

 This looks nice. Is there a thought if this would work with functions as a service (GCP cloud functions, AWS Lambda, etc) for supporting parallel transformation from netcdf to zarr?

I haven't used function as a service before, but yes, I imagine this might be useful for that sort of thing. As long as you can figure out the structure of the overall Zarr datasets ahead of time, you could use region to fill out different parts entirely independently.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   613012939