issue_comments

Table actions

7 rows where issue = 1191575489 sorted by updated_at descending

✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

author_association 2

issue 1

  • Possible race condition writing to zarr store with dask distributed and region kwarg · 7 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1100157468 https://github.com/pydata/xarray/issues/6435#issuecomment-1100157468 https://api.github.com/repos/pydata/xarray/issues/6435 IC_kwDOAMm_X85BkxIc dcherian 2448579 2022-04-15T14:55:43Z 2022-04-15T14:55:43Z MEMBER

It would help if you provided a minimal example that we could run using LocalCluster.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Possible race condition writing to zarr store with dask distributed and region kwarg 1191575489
1099912524 https://github.com/pydata/xarray/issues/6435#issuecomment-1099912524 https://api.github.com/repos/pydata/xarray/issues/6435 IC_kwDOAMm_X85Bj1VM rodgerduffett 23694308 2022-04-15T07:13:33Z 2022-04-15T07:13:33Z NONE

Thank you for looking at this. I understand it is a tricky kind of problem!

Please advise on how to fill out the issue template? When I created this I clicked 'New issue' and it popped up the edit box with '### What is your issue?' There does not seem to be more than this?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Possible race condition writing to zarr store with dask distributed and region kwarg 1191575489
1099668258 https://github.com/pydata/xarray/issues/6435#issuecomment-1099668258 https://api.github.com/repos/pydata/xarray/issues/6435 IC_kwDOAMm_X85Bi5si max-sixty 5635139 2022-04-14T22:07:56Z 2022-04-14T22:07:56Z MEMBER

Hi @rodgerduffett — thanks for the issue. These sorts of issues can be difficult because there are so many moving pieces. Could you try reducing the size of your example a lot? Could you also fill out the issue template? Thanks

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Possible race condition writing to zarr store with dask distributed and region kwarg 1191575489
1099421876 https://github.com/pydata/xarray/issues/6435#issuecomment-1099421876 https://api.github.com/repos/pydata/xarray/issues/6435 IC_kwDOAMm_X85Bh9i0 rodgerduffett 23694308 2022-04-14T17:09:36Z 2022-04-14T17:09:36Z NONE

Updating to latest dask/zarr did not change any behaviour. dask 2022.4.0 pyhd8ed1ab_0 conda-forge dask-core 2022.4.0 pyhd8ed1ab_0 conda-forge dask-glm 0.2.0 py_1 conda-forge dask-jobqueue 0.7.3 pyhd8ed1ab_0 conda-forge dask-labextension 5.2.0 pyhd8ed1ab_0 conda-forge dask-ml 2022.1.22 pyhd8ed1ab_0 conda-forge zarr 2.11.3 pyhd8ed1ab_0 conda-forge xarray 2022.3.0 pyhd8ed1ab_0 conda-forge

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Possible race condition writing to zarr store with dask distributed and region kwarg 1191575489
1099419191 https://github.com/pydata/xarray/issues/6435#issuecomment-1099419191 https://api.github.com/repos/pydata/xarray/issues/6435 IC_kwDOAMm_X85Bh843 rodgerduffett 23694308 2022-04-14T17:07:31Z 2022-04-14T17:07:31Z NONE

Doing: python local_ds=nuvh_ds.compute() local_ds.to_zarr(output_zarrstore,consolidated=True,region={"Time":slice(start_isel,end_isel)}) creates a proper dataset without holes.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Possible race condition writing to zarr store with dask distributed and region kwarg 1191575489
1097960473 https://github.com/pydata/xarray/issues/6435#issuecomment-1097960473 https://api.github.com/repos/pydata/xarray/issues/6435 IC_kwDOAMm_X85BcYwZ rodgerduffett 23694308 2022-04-13T11:52:08Z 2022-04-13T13:39:33Z NONE

Further to this issue the zarr array created is full of holes. The holes appear at random spatial locations throughout the time dimension.

From the zarr store as output to disk: From the dataset before writing to disk: 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Possible race condition writing to zarr store with dask distributed and region kwarg 1191575489
1087406562 https://github.com/pydata/xarray/issues/6435#issuecomment-1087406562 https://api.github.com/repos/pydata/xarray/issues/6435 IC_kwDOAMm_X85A0IHi rodgerduffett 23694308 2022-04-04T10:54:26Z 2022-04-04T10:54:26Z NONE

dask 2022.3.0 pyhd8ed1ab_0 conda-forge dask-core 2022.3.0 pyhd8ed1ab_0 conda-forge dask-glm 0.2.0 py_1 conda-forge dask-jobqueue 0.7.3 pyhd8ed1ab_0 conda-forge dask-labextension 5.2.0 pyhd8ed1ab_0 conda-forge dask-ml 2022.1.22 pyhd8ed1ab_0 conda-forge distributed 2022.3.0 pyhd8ed1ab_0 conda-forge xarray 2022.3.0 pyhd8ed1ab_0 conda-forge

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Possible race condition writing to zarr store with dask distributed and region kwarg 1191575489

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);