issue_comments

Table actions

10 rows where author_association = "MEMBER" and issue = 1385031286 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 3

issue 1

  • open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 · 10 ✖

author_association 1

  • MEMBER · 10 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1492739561 https://github.com/pydata/xarray/issues/7079#issuecomment-1492739561 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85Y-WXp jhamman 2443309 2023-04-01T00:00:24Z 2023-04-01T00:00:24Z MEMBER

@kthyng - any difference when running with parallel=True vs parallel=False?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286
1485339487 https://github.com/pydata/xarray/issues/7079#issuecomment-1485339487 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85YiHtf jhamman 2443309 2023-03-27T15:28:39Z 2023-03-27T15:28:39Z MEMBER

@cefect, @pnorton-usgs, @kthyng - Is this still an issue for you? If so, could you try to run the xarray test suite in #7079 and report back? We haven't been able to trigger the error reported here so we could use some help running the test suite in an "offending" environment.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286
1409716721 https://github.com/pydata/xarray/issues/7079#issuecomment-1409716721 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85UBpHx jhamman 2443309 2023-01-31T03:57:43Z 2023-01-31T03:57:43Z MEMBER

Update: I pushed two new tests to #7488. They are not failing in our test env. If someone that has reported this issue could try running the test suite, that would be super helpful in terms of confirming where the problem lies. 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286
1409358970 https://github.com/pydata/xarray/issues/7079#issuecomment-1409358970 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85UARx6 jhamman 2443309 2023-01-30T21:22:22Z 2023-01-30T23:33:01Z MEMBER

I've opened #7488 which I think has actually exposed a few other failures. I doubt I'll have much time to put into this issue in the near time so anyone should feel free to jump in here.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286
1404127967 https://github.com/pydata/xarray/issues/7079#issuecomment-1404127967 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85TsUrf keewis 14808389 2023-01-25T19:32:12Z 2023-01-25T19:37:14Z MEMBER

iris has the pin in their package metadata

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286
1404118074 https://github.com/pydata/xarray/issues/7079#issuecomment-1404118074 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85TsSQ6 dcherian 2448579 2023-01-25T19:22:52Z 2023-01-25T19:22:52Z MEMBER

o I'm surprised we're not catching this.

Turns out we're running tests on an older working version (logs) even though we don't have a pin. netcdf4 1.6.0 nompi_py310h0a86a1f_103 conda-forge

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286
1404113750 https://github.com/pydata/xarray/issues/7079#issuecomment-1404113750 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85TsRNW jhamman 2443309 2023-01-25T19:18:37Z 2023-01-25T19:18:37Z MEMBER

It would be great if someone could put together a MCVE that reproduces the issue here. We have multiple tests in our test suite that use open_mfdataset with parallel=True, including one that runs against a distributed scheduler and one that runs against the threaded scheduler, so I'm surprised we're not catching this. In any event, the next step would be to develop a test that that triggers the error so we can sort out a fix.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286
1404041288 https://github.com/pydata/xarray/issues/7079#issuecomment-1404041288 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85Tr_hI dcherian 2448579 2023-01-25T18:21:26Z 2023-01-25T19:03:07Z MEMBER

From https://github.com/conda-forge/netcdf4-feedstock/issues/141:

It's on users to manage locking for non-threadsafe resources like netCDF.

@pydata/xarray ~Should we be handling this by default in the netCDF4 backend now?~

EDIT: We already have locks: https://github.com/pydata/xarray/blob/6e77f5e8942206b3e0ab08c3621ade1499d8235b/xarray/backends/netCDF4_.py#L363-L383

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286
1276693638 https://github.com/pydata/xarray/issues/7079#issuecomment-1276693638 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85MGMyG dcherian 2448579 2022-10-12T20:23:11Z 2022-10-12T20:23:11Z MEMBER

My workflow is my own laptop only

Use LocalCluster! ;)

 
{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 1,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286
1276681057 https://github.com/pydata/xarray/issues/7079#issuecomment-1276681057 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85MGJth dcherian 2448579 2022-10-12T20:11:54Z 2022-10-12T20:11:54Z MEMBER

The right fix is to disable threads, like in my example above

This fix will restrict you to serial compute.

You can also parallelize across processes using something like

python PBSCluster( ..., cores=1, processes=2, )

or LocalCluster(threads_per_worker=1, ...)

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);