issue_comments

Table actions

5 rows where author_association = "CONTRIBUTOR" and issue = 1385031286 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

issue 1

  • open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 · 5 ✖

author_association 1

  • CONTRIBUTOR · 5 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1494560788 https://github.com/pydata/xarray/issues/7079#issuecomment-1494560788 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85ZFTAU ocefpaf 950575 2023-04-03T15:44:18Z 2023-04-03T15:44:18Z CONTRIBUTOR

@kthyng those files are on a remote server and that may not be the segfault from the original issue here. It may be a server that is not happy with parallel access. Can you try that with local files?

PS: you can also try with netcdf4<1.6.1 and, if that also fails, it is most likely the server than the issue here.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286
1276668410 https://github.com/pydata/xarray/issues/7079#issuecomment-1276668410 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85MGGn6 ocefpaf 950575 2022-10-12T19:57:35Z 2022-10-12T20:17:08Z CONTRIBUTOR

Note that this is not a bug per se, netcdf-c was never thread safe and, when the work around were removed in netcdf4-python, this issue surfaced. The right fix is to disable threads, like in my example above, or to wait for a netcdf-c release that is thread safe. I don't think the work around will be re-added in netcdf4-python.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286
1276685512 https://github.com/pydata/xarray/issues/7079#issuecomment-1276685512 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85MGKzI ocefpaf 950575 2022-10-12T20:16:41Z 2022-10-12T20:16:41Z CONTRIBUTOR

This fix will restrict you to serial compute.

I was waiting for someone who do stuff on clusters to comment on that. Thanks! (My workflow is my own laptop only, so I'm quite limited on that front :smile:)

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286
1267477522 https://github.com/pydata/xarray/issues/7079#issuecomment-1267477522 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85LjCwS ocefpaf 950575 2022-10-04T19:24:01Z 2022-10-04T19:34:42Z CONTRIBUTOR

Also, you can try:

python import dask dask.config.set(scheduler="single-threaded")

That would ensure you don't use threads when reading with netcdf-c (netcdf4).

Edit: this is not an xarray problem and I recommend to close this issue and follow up with the one already opened upstream.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286
1267159210 https://github.com/pydata/xarray/issues/7079#issuecomment-1267159210 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85Lh1Cq ocefpaf 950575 2022-10-04T15:11:17Z 2022-10-04T15:11:17Z CONTRIBUTOR

I believe you are hitting https://github.com/Unidata/netcdf4-python/issues/1192

The verdict is not out on that one yet. Your parallelization may not be thread safe, which makes 1.6.1 failures that expected. For now, if you can, downgrade to 1.6.0 or use an engine that is thread safe. Maybe h5netcdf (not sure!)?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);