issue_comments

Table actions

3 rows where issue = 613012939 and user = 1197350 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 1

  • rabernat · 3 ✖

issue 1

  • Support parallel writes to regions of zarr stores · 3 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
656637518 https://github.com/pydata/xarray/pull/4035#issuecomment-656637518 https://api.github.com/repos/pydata/xarray/issues/4035 MDEyOklzc3VlQ29tbWVudDY1NjYzNzUxOA== rabernat 1197350 2020-07-10T11:57:40Z 2020-07-10T11:57:40Z MEMBER

Zac, you may be interested in this thread

https://discourse.pangeo.io/t/best-practices-to-go-from-1000s-of-netcdf-files-to-analyses-on-a-hpc-cluster/588/32

Tom White managed to integrate dask with pywren via dask executor. This allows you to read / write zarr with lambda.

Sent from my iPhone

On Jul 9, 2020, at 6:41 PM, Stephan Hoyer notifications@github.com wrote:

 This looks nice. Is there a thought if this would work with functions as a service (GCP cloud functions, AWS Lambda, etc) for supporting parallel transformation from netcdf to zarr?

I haven't used function as a service before, but yes, I imagine this might be useful for that sort of thing. As long as you can figure out the structure of the overall Zarr datasets ahead of time, you could use region to fill out different parts entirely independently.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Support parallel writes to regions of zarr stores 613012939
627318136 https://github.com/pydata/xarray/pull/4035#issuecomment-627318136 https://api.github.com/repos/pydata/xarray/issues/4035 MDEyOklzc3VlQ29tbWVudDYyNzMxODEzNg== rabernat 1197350 2020-05-12T12:42:12Z 2020-05-12T12:42:37Z MEMBER

A similar neat feature would be to read xarray datasets from regions of zarr groups w/o dask arrays.

@nbren12 - this has always been supported. Just call open_zarr(..., chunks=False) and then subset using sel / isel. 

{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 1
}
   Support parallel writes to regions of zarr stores 613012939
625865523 https://github.com/pydata/xarray/pull/4035#issuecomment-625865523 https://api.github.com/repos/pydata/xarray/issues/4035 MDEyOklzc3VlQ29tbWVudDYyNTg2NTUyMw== rabernat 1197350 2020-05-08T15:16:54Z 2020-05-08T15:16:54Z MEMBER

Stephan, this seems like a great addition. Thanks for getting it started!

I'm curious how this interacts with dimension coordinates. Your example bypasses this. But what if dimension coordinates are present. How do we handle alignment issues? For example, what if I call ds.to_zarr(path , region=selection), but the dimension coordinates of ds don't align with the dimension coordinates of the store at path"

  1. Officially document that the compute argument only controls writing array values, not metadata (at least for zarr).

:+1:

4. Like (2), but raise an error instead of a warning. Require the user to explicitly drop them with .drop(). This is probably the safest behavior.

:+1:

I think only advanced users will want to use this feature.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Support parallel writes to regions of zarr stores 613012939

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);