issue_comments

Table actions

4 rows where issue = 1385031286 and user = 2448579 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

user 1

  • dcherian · 4 ✖

issue 1

  • open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 · 4 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1404118074 https://github.com/pydata/xarray/issues/7079#issuecomment-1404118074 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85TsSQ6 dcherian 2448579 2023-01-25T19:22:52Z 2023-01-25T19:22:52Z MEMBER

o I'm surprised we're not catching this.

Turns out we're running tests on an older working version (logs) even though we don't have a pin. netcdf4 1.6.0 nompi_py310h0a86a1f_103 conda-forge

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286
1404041288 https://github.com/pydata/xarray/issues/7079#issuecomment-1404041288 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85Tr_hI dcherian 2448579 2023-01-25T18:21:26Z 2023-01-25T19:03:07Z MEMBER

From https://github.com/conda-forge/netcdf4-feedstock/issues/141:

It's on users to manage locking for non-threadsafe resources like netCDF.

@pydata/xarray ~Should we be handling this by default in the netCDF4 backend now?~

EDIT: We already have locks: https://github.com/pydata/xarray/blob/6e77f5e8942206b3e0ab08c3621ade1499d8235b/xarray/backends/netCDF4_.py#L363-L383

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286
1276693638 https://github.com/pydata/xarray/issues/7079#issuecomment-1276693638 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85MGMyG dcherian 2448579 2022-10-12T20:23:11Z 2022-10-12T20:23:11Z MEMBER

My workflow is my own laptop only

Use LocalCluster! ;)

 
{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 1,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286
1276681057 https://github.com/pydata/xarray/issues/7079#issuecomment-1276681057 https://api.github.com/repos/pydata/xarray/issues/7079 IC_kwDOAMm_X85MGJth dcherian 2448579 2022-10-12T20:11:54Z 2022-10-12T20:11:54Z MEMBER

The right fix is to disable threads, like in my example above

This fix will restrict you to serial compute.

You can also parallelize across processes using something like

python PBSCluster( ..., cores=1, processes=2, )

or LocalCluster(threads_per_worker=1, ...)

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   open_mfdataset parallel=True failing with netcdf4 >= 1.6.1 1385031286

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);