issue_comments

Table actions

6 rows where issue = 274308380 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 3

author_association 2

issue 1

  • Possible regression with PyNIO data not being lazily loaded · 6 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
345310488 https://github.com/pydata/xarray/issues/1720#issuecomment-345310488 https://api.github.com/repos/pydata/xarray/issues/1720 MDEyOklzc3VlQ29tbWVudDM0NTMxMDQ4OA== WeatherGod 291576 2017-11-17T17:33:13Z 2017-11-17T17:33:13Z CONTRIBUTOR

Awesome! Thanks!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Possible regression with PyNIO data not being lazily loaded 274308380
345157498 https://github.com/pydata/xarray/issues/1720#issuecomment-345157498 https://api.github.com/repos/pydata/xarray/issues/1720 MDEyOklzc3VlQ29tbWVudDM0NTE1NzQ5OA== fujiisoup 6815844 2017-11-17T06:32:14Z 2017-11-17T06:32:14Z MEMBER

OK. Found a bug and sent a PR. This was the smallest bug fix I have ever done :)

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Possible regression with PyNIO data not being lazily loaded 274308380
345126212 https://github.com/pydata/xarray/issues/1720#issuecomment-345126212 https://api.github.com/repos/pydata/xarray/issues/1720 MDEyOklzc3VlQ29tbWVudDM0NTEyNjIxMg== fujiisoup 6815844 2017-11-17T02:23:09Z 2017-11-17T02:23:59Z MEMBER

I don't think we have tests of this case for any backend other than dask after #1532. In my local tests, many backend array are loaded after print.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Possible regression with PyNIO data not being lazily loaded 274308380
345125593 https://github.com/pydata/xarray/issues/1720#issuecomment-345125593 https://api.github.com/repos/pydata/xarray/issues/1720 MDEyOklzc3VlQ29tbWVudDM0NTEyNTU5Mw== shoyer 1217238 2017-11-17T02:18:55Z 2017-11-17T02:18:55Z MEMBER

The private Variable._in_memory attribute (e.g., ds.variables['temperature']._in_memory) will tell you if xarray thinks the data is in memory for the purposes of printing array values.

Are you using dask to load this data? That can also change things.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Possible regression with PyNIO data not being lazily loaded 274308380
345124033 https://github.com/pydata/xarray/issues/1720#issuecomment-345124033 https://api.github.com/repos/pydata/xarray/issues/1720 MDEyOklzc3VlQ29tbWVudDM0NTEyNDAzMw== WeatherGod 291576 2017-11-17T02:08:50Z 2017-11-17T02:08:50Z CONTRIBUTOR

Is there a convenient sentinel I can check for loaded-ness? The only reason I noticed this was I was debugging another problem with my processing of HRRR files (~600mb each) and the memory usage shot up (did you know that top will report memory usage as fractions of terabytes when you get high enough?). I could test this with some smaller netcdf4 files if I could just loop through the variables and assert some sentinal.

On Thu, Nov 16, 2017 at 8:57 PM, Stephan Hoyer notifications@github.com wrote:

@WeatherGod https://github.com/weathergod can you verify that you don't get immediate loading when loading netCDF files, e.g., with scipy or netCDF4-python?

We did change how loading of data works with printing in this release (

1532 https://github.com/pydata/xarray/pull/1532), but if anything the

changes should go the other way, to do less loading of data.

I'm having trouble debugging this locally because I can't seem to get a working version of pynio installed from conda-forge on OS X (running into various ABI incompatibility issues when I try this in a new conda environment).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/1720#issuecomment-345122204, or mute the thread https://github.com/notifications/unsubscribe-auth/AARy-MO7la8KSJnQoto8Kso5gBYedUKQks5s3OgSgaJpZM4Qflk- . 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Possible regression with PyNIO data not being lazily loaded 274308380
345122204 https://github.com/pydata/xarray/issues/1720#issuecomment-345122204 https://api.github.com/repos/pydata/xarray/issues/1720 MDEyOklzc3VlQ29tbWVudDM0NTEyMjIwNA== shoyer 1217238 2017-11-17T01:57:37Z 2017-11-17T01:57:37Z MEMBER

@WeatherGod can you verify that you don't get immediate loading when loading netCDF files, e.g., with scipy or netCDF4-python?

We did change how loading of data works with printing in this release (#1532), but if anything the changes should go the other way, to do less loading of data.

I'm having trouble debugging this locally because I can't seem to get a working version of pynio installed from conda-forge on OS X (running into various ABI incompatibility issues when I try this in a new conda environment).

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Possible regression with PyNIO data not being lazily loaded 274308380

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);