issue_comments

Table actions

3 rows where issue = 168936861 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 2

author_association 2

issue 1

  • PyNIO backend doesn't play well with open_mfdataset · 3 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
239005082 https://github.com/pydata/xarray/issues/936#issuecomment-239005082 https://api.github.com/repos/pydata/xarray/issues/936 MDEyOklzc3VlQ29tbWVudDIzOTAwNTA4Mg== shoyer 1217238 2016-08-10T21:08:39Z 2016-08-10T21:08:39Z MEMBER

The fix I posted on stackoverflow is a bandaid solution -- it requires loading every file into memory all at once, which can be problematic if you have large amounts of data.

It occurs to me that the problem might be that we were attempting to concurrently load data from multiple variables in a single file at once. If this is the issue, then it's something we can work around pretty easily with xarray. I'll run some tests later to verify.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   PyNIO backend doesn't play well with open_mfdataset 168936861
239002748 https://github.com/pydata/xarray/issues/936#issuecomment-239002748 https://api.github.com/repos/pydata/xarray/issues/936 MDEyOklzc3VlQ29tbWVudDIzOTAwMjc0OA== david-ian-brown 2020717 2016-08-10T21:00:22Z 2016-08-10T21:00:22Z NONE

Hi Stephan, I notice that you posted a solution of sorts on stackoverflow. Is this a real fix or a bandaid for the problem? I ran a few tests with pynio by itself where there was no problem opening e.g. a GRIB file for reading and a NetCDF file for writing at the same time. I am wondering if the issue involves using pynio and the netCDF4 modules at the same time. Let me know if still think something in pynio is not working correctly. Thanks. -dave

On Tue, Aug 2, 2016 at 1:35 PM, David Brown dbrown@ucar.edu wrote:

Hi Stephan,

I will look into this issue. -dave

On Tue, Aug 2, 2016 at 11:01 AM, Stephan Hoyer notifications@github.com wrote:

As reported on StackOverflow: http://stackoverflow.com/ questions/38711915/segmentation-fault-writing-xarray-datset-to-netcdf-or- dataframe/

It appears that we can only open a single file at a time with pynio?

Adding a thread lock via lock=True didn't solve the issue.

cc @david-ian-brown https://github.com/david-ian-brown

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/936, or mute the thread https://github.com/notifications/unsubscribe-auth/AB7VbedP7yIJsI6DEz0PLOnVJJBDadaKks5qb3gDgaJpZM4Ja1AT .

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   PyNIO backend doesn't play well with open_mfdataset 168936861
237017929 https://github.com/pydata/xarray/issues/936#issuecomment-237017929 https://api.github.com/repos/pydata/xarray/issues/936 MDEyOklzc3VlQ29tbWVudDIzNzAxNzkyOQ== david-ian-brown 2020717 2016-08-02T19:35:32Z 2016-08-02T19:35:32Z NONE

Hi Stephan,

I will look into this issue. -dave

On Tue, Aug 2, 2016 at 11:01 AM, Stephan Hoyer notifications@github.com wrote:

As reported on StackOverflow: http://stackoverflow.com/questions/38711915/segmentation-fault-writing-xarray-datset-to-netcdf-or-dataframe/

It appears that we can only open a single file at a time with pynio?

Adding a thread lock via lock=True didn't solve the issue.

cc @david-ian-brown https://github.com/david-ian-brown

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/936, or mute the thread https://github.com/notifications/unsubscribe-auth/AB7VbedP7yIJsI6DEz0PLOnVJJBDadaKks5qb3gDgaJpZM4Ja1AT .

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   PyNIO backend doesn't play well with open_mfdataset 168936861

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);