issue_comments

Table actions

4 rows where issue = 91547750 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 2

author_association 2

issue 1

  • Preprocess argument for open_mfdataset and threading lock · 4 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
118097399 https://github.com/pydata/xarray/pull/446#issuecomment-118097399 https://api.github.com/repos/pydata/xarray/issues/446 MDEyOklzc3VlQ29tbWVudDExODA5NzM5OQ== razcore-rad 1177508 2015-07-02T17:16:41Z 2015-07-02T17:16:41Z NONE

I need to get my head around this... I know that when you do list comprehension, this isn't lazy so basically it goes through the loop and evaluates for each iteration... so I thought that:

if preprocess is not None: datasets = [preprocess(ds) for ds in datasets]

translates to forcing the application of the preprocess function to the dataset effectively loading it in memory... anyway, this is really cool, I'll definitely try it out :+1: 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Preprocess argument for open_mfdataset and threading lock 91547750
118094621 https://github.com/pydata/xarray/pull/446#issuecomment-118094621 https://api.github.com/repos/pydata/xarray/issues/446 MDEyOklzc3VlQ29tbWVudDExODA5NDYyMQ== shoyer 1217238 2015-07-02T17:09:12Z 2015-07-02T17:09:12Z MEMBER

Nope, each dataset is loaded lazily when using open_mfdataset (via dask). As long as you stick to xray operations and don't actually manually load data into memory (e.g., by calling .load()) the data is only accessed and transformed by preprocess on a need only basis.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Preprocess argument for open_mfdataset and threading lock 91547750
118093087 https://github.com/pydata/xarray/pull/446#issuecomment-118093087 https://api.github.com/repos/pydata/xarray/issues/446 MDEyOklzc3VlQ29tbWVudDExODA5MzA4Nw== razcore-rad 1177508 2015-07-02T17:00:47Z 2015-07-02T17:00:47Z NONE

I have a question about this preprocess thing. Would it mean now that... basically xray will load all data in memory? because of the preprocesing step, whereas before... or at least that's what I understood from the documentation, xray would access the data by a need only basis.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Preprocess argument for open_mfdataset and threading lock 91547750
116399654 https://github.com/pydata/xarray/pull/446#issuecomment-116399654 https://api.github.com/repos/pydata/xarray/issues/446 MDEyOklzc3VlQ29tbWVudDExNjM5OTY1NA== shoyer 1217238 2015-06-29T03:01:06Z 2015-06-29T03:01:06Z MEMBER

Going to merge this shortly, unless anyone has a better name for preprocess

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Preprocess argument for open_mfdataset and threading lock 91547750

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);