issue_comments

Table actions

3 rows where issue = 950882492 and user = 35968931 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 1

  • TomNicholas · 3 ✖

issue 1

  • Polyfit performance on large datasets - Suboptimal dask task graph · 3 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
961349927 https://github.com/pydata/xarray/issues/5629#issuecomment-961349927 https://api.github.com/repos/pydata/xarray/issues/5629 IC_kwDOAMm_X845TQkn TomNicholas 35968931 2021-11-04T19:25:41Z 2021-11-04T19:26:51Z MEMBER

I was thinking the general idea of reshape_block seems like a clever trick to get around the apply_ufunc constraints and could be used for some other functions as well. But maybe there aren't that many functions that could make use of it.

The idea seems similar to what we used in xhistogram, which uses blockwise to apply a function that reshapes the data in that block to be 1D.

EDIT: But in that case our algorithm was one that could be applied blockwise, apply_ufunc was not used.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Polyfit performance on large datasets - Suboptimal dask task graph 950882492
961187488 https://github.com/pydata/xarray/issues/5629#issuecomment-961187488 https://api.github.com/repos/pydata/xarray/issues/5629 IC_kwDOAMm_X845So6g TomNicholas 35968931 2021-11-04T16:04:17Z 2021-11-04T16:04:17Z MEMBER

The problem is that there is no general solution here. You have to write a chunk-aware function and then use apply_ufunc(..., dask="allowed"). This suggestion only works because dask's lstsq can work with chunked dimensions.

Sorry, what case does your suggestion not handle? It would be really nice to support chunked core dims if we are going to the effort of rewriting polyfit.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Polyfit performance on large datasets - Suboptimal dask task graph 950882492
885087800 https://github.com/pydata/xarray/issues/5629#issuecomment-885087800 https://api.github.com/repos/pydata/xarray/issues/5629 IC_kwDOAMm_X840wV44 TomNicholas 35968931 2021-07-22T17:31:02Z 2021-07-22T17:31:58Z MEMBER

Thanks for the clear example @jbusecke . I think your example is helpful enough that we should keep this open (even if the same PR will hopefully close both this and #4554)

Was about to say the same @dcherian! @aulemahal what do you think about how easy it might be to change polyfit to use apply_ufunc?

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Polyfit performance on large datasets - Suboptimal dask task graph 950882492

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);