issue_comments

Table actions

3 rows where author_association = "MEMBER", issue = 627600168 and user = 2448579 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 1

  • dcherian · 3 ✖

issue 1

  • Unexpected chunking behavior when using `xr.align` with `join='outer'` · 3 ✖

author_association 1

  • MEMBER · 3 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
643513541 https://github.com/pydata/xarray/issues/4112#issuecomment-643513541 https://api.github.com/repos/pydata/xarray/issues/4112 MDEyOklzc3VlQ29tbWVudDY0MzUxMzU0MQ== dcherian 2448579 2020-06-12T22:55:12Z 2020-06-12T22:55:12Z MEMBER

One option might be to rewrite Dask's indexing functionality to "split" chunks that are much larger than their inputs into smaller pieces, even if they all come from the same input chunk?

This is Tom's proposed solution in https://github.com/dask/dask/issues/6270

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Unexpected chunking behavior when using `xr.align` with `join='outer'` 627600168
643346497 https://github.com/pydata/xarray/issues/4112#issuecomment-643346497 https://api.github.com/repos/pydata/xarray/issues/4112 MDEyOklzc3VlQ29tbWVudDY0MzM0NjQ5Nw== dcherian 2448579 2020-06-12T15:51:31Z 2020-06-12T15:52:58Z MEMBER

Thanks @TomAugspurger

I think an upstream dask solution would be useful.

xarray automatic aligns objects everywhere and this alignment is what is blowing things up. For this reason I think xarray should explicitly chunk the indexer when aligning. We could use a reasonable chunk size like median chunk size of dataarray along that axis — this would respect the user's chunksize choices.

@shoyer What do you think?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Unexpected chunking behavior when using `xr.align` with `join='outer'` 627600168
636334010 https://github.com/pydata/xarray/issues/4112#issuecomment-636334010 https://api.github.com/repos/pydata/xarray/issues/4112 MDEyOklzc3VlQ29tbWVudDYzNjMzNDAxMA== dcherian 2448579 2020-05-30T13:52:33Z 2020-05-30T13:53:31Z MEMBER

Great diagnosis @jbusecke .

Ultimately this comes down to dask indexing

``` python import dask.array

arr = dask.array.from_array([0, 1, 2, 3], chunks=(1,)) print(arr.chunks) # ((1, 1, 1, 1),)

align calls reindex which indexes with something like this

indexer = [0, 1, 2, 3, ] + [-1,] * 111 print(arr[indexer].chunks) # ((1, 1, 1, 112),)

maybe something like this is a solution

lazy_indexer = dask.array.from_array(indexer, chunks=arr.chunks[0][0], name="idx") print(arr[lazy_indexer].chunks) # ((1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1),) ```

cc @TomAugspurger, the issue here is that big 112 size chunk takes down the cluster in https://github.com/NCAR/intake-esm/issues/225

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Unexpected chunking behavior when using `xr.align` with `join='outer'` 627600168

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);