issue_comments

Table actions

6 rows where author_association = "NONE" and user = 6948919 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: issue_url, created_at (date), updated_at (date)

user 1

  • bekatd · 6 ✖

author_association 1

  • NONE · 6 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
778972571 https://github.com/pydata/xarray/issues/3961#issuecomment-778972571 https://api.github.com/repos/pydata/xarray/issues/3961 MDEyOklzc3VlQ29tbWVudDc3ODk3MjU3MQ== bekatd 6948919 2021-02-15T06:11:34Z 2021-02-15T06:11:34Z NONE

Please make some dummy tests, I did time.sleep, prior every operation. This was the only workaround that really worked. 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Hangs while saving netcdf file opened using xr.open_mfdataset with lock=None 597657663
778839471 https://github.com/pydata/xarray/issues/3961#issuecomment-778839471 https://api.github.com/repos/pydata/xarray/issues/3961 MDEyOklzc3VlQ29tbWVudDc3ODgzOTQ3MQ== bekatd 6948919 2021-02-14T20:47:27Z 2021-02-14T20:47:27Z NONE

Is the current recommended solution to set lock=False and retry until success? Or, is it to keep lock=None and use zarr instead? @dcherian

Or alternatively you can try to set sleep between openings.

When you try to open same file from different functions with different operations, it is better to keep file opening function wrapped with a 1 second delay/sleep rather than direct open

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Hangs while saving netcdf file opened using xr.open_mfdataset with lock=None 597657663
730033698 https://github.com/pydata/xarray/issues/3961#issuecomment-730033698 https://api.github.com/repos/pydata/xarray/issues/3961 MDEyOklzc3VlQ29tbWVudDczMDAzMzY5OA== bekatd 6948919 2020-11-19T00:02:09Z 2020-11-19T00:04:13Z NONE

I have the same behaviour with MacOS (10.15). xarray=0.16.1, dask=2.30.0, netcdf4=1.5.4. Sometimes saves, sometimes doesn't. lock=False seems to work.

Lock false sometimes throws hd5 error. No clear solution.

The only solution I have found, sleep method for 1 second

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Hangs while saving netcdf file opened using xr.open_mfdataset with lock=None 597657663
690922236 https://github.com/pydata/xarray/issues/4406#issuecomment-690922236 https://api.github.com/repos/pydata/xarray/issues/4406 MDEyOklzc3VlQ29tbWVudDY5MDkyMjIzNg== bekatd 6948919 2020-09-11T07:18:03Z 2020-09-11T07:18:03Z NONE

Did this work reliably in the past? If so, any clues about specific versions of dask and/or netCDF that cause the issue would be helpful.

@TomAugspurger do you know off-hand if there have been any recent changes in Dask's scheduler that could have caused this?

I am new to xarray dask thing but month ago it was woking without issues. I recently reinstalled python and dont know if versions differs from previous one

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Threading Lock issue with to_netcdf and Dask arrays 694112301
690340047 https://github.com/pydata/xarray/issues/4406#issuecomment-690340047 https://api.github.com/repos/pydata/xarray/issues/4406 MDEyOklzc3VlQ29tbWVudDY5MDM0MDA0Nw== bekatd 6948919 2020-09-10T14:48:38Z 2020-09-10T14:48:38Z NONE

Using:

  • xarray=0.16.0
  • dask=2.25.0
  • netcdf4=1.5.4

I am experiencing same when trying to write netcdf file using to_netcdf() on a files opened via xr.open_mfdataset with lock=None (which is default).

Then I tried to open files with lock=False and it worked like a charm. Issue have been gone for 100% of times.

BUT

Now I am facing different issue. Seems that hdf5 IS NOT thread safe, since I encounter NetCDF: HDF error while applying different function on a netcdf files which were previously processed by another functions with lock=False. Script just terminates not even reaching any calculation step in the code. seems like lock=False works opposite and file is in a corrupted mode?

This is the BIGGEST issue and needs resolve ASAP

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Threading Lock issue with to_netcdf and Dask arrays 694112301
690332310 https://github.com/pydata/xarray/issues/3961#issuecomment-690332310 https://api.github.com/repos/pydata/xarray/issues/3961 MDEyOklzc3VlQ29tbWVudDY5MDMzMjMxMA== bekatd 6948919 2020-09-10T14:36:53Z 2020-09-10T14:38:51Z NONE

Using:

  • xarray=0.16.0
  • dask=2.25.0
  • netcdf4=1.5.4

I am experiencing same when trying to write netcdf file using xr.to_netcdf() on a files opened via xr.open_mfdataset with lock=None.

Then I tried OP's suggestion and it worked like a charm

BUT

Now I am facing different issue. Seems that hdf5 IS NOT thread safe, since I encounter NetCDF: HDF error while applying different function on a netcdf file, previously were processed by another function with lock=False. script just terminates not even reaching any calculation step in the code. seems like lock=False works opposite and file is in a corrupted mode?

This is the BIGGEST issue and needs resolve ASAP

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Hangs while saving netcdf file opened using xr.open_mfdataset with lock=None 597657663

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);