issue_comments

Table actions

4 rows where author_association = "MEMBER" and issue = 1064837571 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 3

issue 1

  • Threadlocking in DataArray calculations for zarr data depending on where it's loaded from (S3 vs local) · 4 ✖

author_association 1

  • MEMBER · 4 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1017189009 https://github.com/pydata/xarray/issues/6033#issuecomment-1017189009 https://api.github.com/repos/pydata/xarray/issues/6033 IC_kwDOAMm_X848oRKR jhamman 2443309 2022-01-20T07:25:28Z 2022-01-20T19:59:22Z MEMBER

It is worth mentioning that, specifically when using Zarr with fsspec, you have multiple layers of caching available.

  1. You can ask fsspec to cache locally:

python path = 's3://hrrrzarr/sfc/20211124/20211124_00z_fcst.zarr/surface/PRES' ds = xr.open_zarr('simplecache::'+path) (more details on configuration: https://filesystem-spec.readthedocs.io/en/latest/features.html#caching-files-locally)

  1. You can ask Zarr to cache chunks as they are read:

python mapper = fsspec.get_mapper(path) store = LRUStoreCache(mapper, max_size=1e9) ds = xr.open_zarr(store)

(more details on configuration here: https://zarr.readthedocs.io/en/stable/api/storage.html#zarr.storage.LRUStoreCache)

  1. Configure a more complex mapper/cache using 3rd party mappers (i.e. Zict)

perhaps @martindurant has more to add here?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Threadlocking in DataArray calculations for zarr data depending on where it's loaded from (S3 vs local) 1064837571
981064185 https://github.com/pydata/xarray/issues/6033#issuecomment-981064185 https://api.github.com/repos/pydata/xarray/issues/6033 IC_kwDOAMm_X846edn5 Illviljan 14371165 2021-11-28T10:59:05Z 2021-11-28T10:59:05Z MEMBER

If you think the data would fit in memory maybe #5704 would be enough?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Threadlocking in DataArray calculations for zarr data depending on where it's loaded from (S3 vs local) 1064837571
980501470 https://github.com/pydata/xarray/issues/6033#issuecomment-980501470 https://api.github.com/repos/pydata/xarray/issues/6033 IC_kwDOAMm_X846cUPe max-sixty 5635139 2021-11-27T04:43:31Z 2021-11-27T04:43:31Z MEMBER

Is there a way to check what is and isn't downloaded?

What is the time difference between the approach you've tried vs. before anything is downloaded? 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Threadlocking in DataArray calculations for zarr data depending on where it's loaded from (S3 vs local) 1064837571
980459050 https://github.com/pydata/xarray/issues/6033#issuecomment-980459050 https://api.github.com/repos/pydata/xarray/issues/6033 IC_kwDOAMm_X846cJ4q max-sixty 5635139 2021-11-26T22:41:32Z 2021-11-26T22:41:32Z MEMBER

Thanks @adair-kovac .

To what extent is this the time to download the data? How big is the dataset? What's the absolute difference for a very small dataset? Or for a large dataset including the time to download the data first?

The threading issue may be threading contention, or it could be the main thread waiting for another thread to complete the download (others will know more here).

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Threadlocking in DataArray calculations for zarr data depending on where it's loaded from (S3 vs local) 1064837571

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);