issue_comments

Table actions

8 rows where issue = 252358450 and user = 1217238 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 1

  • shoyer · 8 ✖

issue 1

  • Automatic parallelization for dask arrays in apply_ufunc · 8 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
335316764 https://github.com/pydata/xarray/pull/1517#issuecomment-335316764 https://api.github.com/repos/pydata/xarray/issues/1517 MDEyOklzc3VlQ29tbWVudDMzNTMxNjc2NA== shoyer 1217238 2017-10-09T23:28:52Z 2017-10-09T23:28:52Z MEMBER

I'll start on my PR to expose this as public API -- hopefully will make some progress on my flight from NY to SF tonight.

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Automatic parallelization for dask arrays in apply_ufunc 252358450
335315831 https://github.com/pydata/xarray/pull/1517#issuecomment-335315831 https://api.github.com/repos/pydata/xarray/issues/1517 MDEyOklzc3VlQ29tbWVudDMzNTMxNTgzMQ== shoyer 1217238 2017-10-09T23:22:33Z 2017-10-09T23:22:33Z MEMBER

I think this is ready.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Automatic parallelization for dask arrays in apply_ufunc 252358450
330701517 https://github.com/pydata/xarray/pull/1517#issuecomment-330701517 https://api.github.com/repos/pydata/xarray/issues/1517 MDEyOklzc3VlQ29tbWVudDMzMDcwMTUxNw== shoyer 1217238 2017-09-19T23:25:08Z 2017-09-19T23:25:08Z MEMBER

I have a design question here: how should we handle cases where a core dimension exists in multiple chunks? For example, suppose you are applying a function that needs access to every point along the "time" axis at once (e.g., an auto-correlation function).

Should we: 1. Automatically rechunk along "time" into a single chunk, or 2. Raise an error, and require the user to rechunk manually (xref https://github.com/dask/dask/issues/2689 for API on this)

Currently we do behavior 1, but behavior 2 might be more user friendly. Otherwise it could be pretty easy to inadvertently pass in a dask array (e.g., in small chunks along time) that apply_ufunc would load into memory by putting in a single chunk.

dask.array has some heuristics to protect against this in rechunk() but I'm not sure they are effective enough to catch this. (@mrocklin?)

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Automatic parallelization for dask arrays in apply_ufunc 252358450
330022743 https://github.com/pydata/xarray/pull/1517#issuecomment-330022743 https://api.github.com/repos/pydata/xarray/issues/1517 MDEyOklzc3VlQ29tbWVudDMzMDAyMjc0Mw== shoyer 1217238 2017-09-17T05:39:38Z 2017-09-17T05:39:38Z MEMBER

Alternatively apply_ufunc could see if the func object has a pre_dask_atop method, and apply it if it does.

This seems like a reasonable option to me. Once we get this merged, want to make a PR?

@jhamman could you give this a review? I have not included extensive documentation yet, but I am also reluctant to squeeze that into this PR before we make it public API. (Which I'd like to save for another one.)

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Automatic parallelization for dask arrays in apply_ufunc 252358450
328299112 https://github.com/pydata/xarray/pull/1517#issuecomment-328299112 https://api.github.com/repos/pydata/xarray/issues/1517 MDEyOklzc3VlQ29tbWVudDMyODI5OTExMg== shoyer 1217238 2017-09-09T19:38:09Z 2017-09-09T19:38:09Z MEMBER

@nbren12 Probably the best way to do ghosting with the current interface is to write a function that acts on dask array objects to apply the ghosting, and then apply it using apply_ufunc. I don't see an easy way to incorporate it into the current interface, which is already getting pretty complicated.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Automatic parallelization for dask arrays in apply_ufunc 252358450
324734974 https://github.com/pydata/xarray/pull/1517#issuecomment-324734974 https://api.github.com/repos/pydata/xarray/issues/1517 MDEyOklzc3VlQ29tbWVudDMyNDczNDk3NA== shoyer 1217238 2017-08-24T19:34:49Z 2017-08-24T19:34:49Z MEMBER

@mrocklin I split that discussion off to #1525.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Automatic parallelization for dask arrays in apply_ufunc 252358450
324732200 https://github.com/pydata/xarray/pull/1517#issuecomment-324732200 https://api.github.com/repos/pydata/xarray/issues/1517 MDEyOklzc3VlQ29tbWVudDMyNDczMjIwMA== shoyer 1217238 2017-08-24T19:22:56Z 2017-08-24T19:22:56Z MEMBER

@mrocklin Yes, that took a few seconds (due to hashing the array contents). Would you suggest setting name=False by default for xarray's chunk() method?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Automatic parallelization for dask arrays in apply_ufunc 252358450
324705244 https://github.com/pydata/xarray/pull/1517#issuecomment-324705244 https://api.github.com/repos/pydata/xarray/issues/1517 MDEyOklzc3VlQ29tbWVudDMyNDcwNTI0NA== shoyer 1217238 2017-08-24T17:38:29Z 2017-08-24T17:38:29Z MEMBER

What's rs.randn()?

Oops, fixed.

When this makes it into the public facing API it would be nice to include some guidance on how the chunking scheme affects the run time.

We already have some tips here: http://xarray.pydata.org/en/stable/dask.html#chunking-and-performance

More ambitiously I could imagine an API such as array1.chunk('place') or array1.chunk('auto') meaning to figure out a reasonable chunking scheme only once .compute() is called so that all the compute steps are known.

Yes, this would be great.

Maybe this is more specific to dask than xarray. I believe it would also be difficult.

I agree with both!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Automatic parallelization for dask arrays in apply_ufunc 252358450

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);