issue_comments

Table actions

3 rows where author_association = "MEMBER" and issue = 493058488 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

issue 1

  • `ds.load()` with local files stalls and fails, and `to_zarr` does not include `store` in the dask graph · 3 ✖

author_association 1

  • MEMBER · 3 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
531499393 https://github.com/pydata/xarray/issues/3306#issuecomment-531499393 https://api.github.com/repos/pydata/xarray/issues/3306 MDEyOklzc3VlQ29tbWVudDUzMTQ5OTM5Mw== rabernat 1197350 2019-09-14T17:47:10Z 2019-09-14T17:47:10Z MEMBER

What if you just use a dask local cluster, rather than a distributed cluster? Then you can just write to a local directory.

And what if you don’t use a distributed cluster at all, just the threaded scheduler?

In my experience with these problems, by systematically removing layers of complexity from the scenario, we often come to the root of the issue

On Sep 14, 2019, at 11:03 AM, Aimee Barciauskas notifications@github.com wrote:

@rabernat good points. One thing I'm not sure of how to make reproducible is calling a remote file store, since I think it usually requires calling to a write-protected cloud storage provider. Any tips on this?

I have what should be an otherwise working example here: https://gist.github.com/abarciauskas-bgse/d0aac2ae9bf0b06f52a577d0a6251b2d - let me know if this is an ok format to share for reproducing the issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   `ds.load()` with local files stalls and fails, and `to_zarr` does not include `store` in the dask graph 493058488
531489772 https://github.com/pydata/xarray/issues/3306#issuecomment-531489772 https://api.github.com/repos/pydata/xarray/issues/3306 MDEyOklzc3VlQ29tbWVudDUzMTQ4OTc3Mg== rabernat 1197350 2019-09-14T15:44:32Z 2019-09-14T15:44:32Z MEMBER

Does the problem only arise when writing to s3fs? Or can you reproduce it writing to a local Zarr directory store?

Sent from my iPhone

On Sep 14, 2019, at 11:03 AM, Aimee Barciauskas notifications@github.com wrote:

@rabernat good points. One thing I'm not sure of how to make reproducible is calling a remote file store, since I think it usually requires calling to a write-protected cloud storage provider. Any tips on this?

I have what should be an otherwise working example here: https://gist.github.com/abarciauskas-bgse/d0aac2ae9bf0b06f52a577d0a6251b2d - let me know if this is an ok format to share for reproducing the issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   `ds.load()` with local files stalls and fails, and `to_zarr` does not include `store` in the dask graph 493058488
531437477 https://github.com/pydata/xarray/issues/3306#issuecomment-531437477 https://api.github.com/repos/pydata/xarray/issues/3306 MDEyOklzc3VlQ29tbWVudDUzMTQzNzQ3Nw== rabernat 1197350 2019-09-14T02:00:54Z 2019-09-14T02:00:54Z MEMBER

@aidanheerdegen - thanks so much for posting this issue! I think a lot of people run into these sorts of problems, so it's useful to have an example on the issue tracker.

These problems can unfortunately be very hard to debug. If other developers can quickly reproduce your exact same error on their own systems, then we can try to dig deeper. However, I can't run the code you shared. If I paste it into a notebook, I get FileNotFoundError: [Errno 2] No such file or directory: b'/home/jovyan/cmip6-bot/2018/153/20180602090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc' Your example requires your files, which I don't have. I noticed some later examples point to a podaac opendap sever, commented out with the comment ```

This works fine

base_url = 'https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/JPL/MUR/v4.1/'

``` but I couldn't tell if I was supposed to run that part or not.

So unfortunately I have to ask you for some tweaks to your question. Could you either: 1. edit this example so that it points to an opendap server or other globally accessible endpoint on the internet? OR 1. replace the real data with synthetically generated data (e.g. use dask.random.random to create arrays instead of loading from disk)

Although 2 is a pain, it actually usually helps surface bugs by removing part of the I/O.

Thanks again for your contribution.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   `ds.load()` with local files stalls and fails, and `to_zarr` does not include `store` in the dask graph 493058488

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);