issue_comments

Table actions

5 rows where issue = 929818771 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 3

issue 1

  • Very poor html repr performance on large multi-indexes · 5 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1075109337 https://github.com/pydata/xarray/issues/5529#issuecomment-1075109337 https://api.github.com/repos/pydata/xarray/issues/5529 IC_kwDOAMm_X85AFN3Z benbovy 4160723 2022-03-22T12:23:23Z 2022-03-22T12:23:23Z MEMBER

But weirdly the linked PR is attempting to do that — so maybe this code path doesn't hit that change?

I think the linked PR only fixed the summary (inline) repr. The bottleneck here is when formatting the array detailed view for the multi-index coordinates, which triggers the conversion of the whole pandas MultiIndex (tuple elements) and each of its levels as a numpy arrays.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Very poor html repr performance on large multi-indexes 929818771
868839990 https://github.com/pydata/xarray/issues/5529#issuecomment-868839990 https://api.github.com/repos/pydata/xarray/issues/5529 MDEyOklzc3VlQ29tbWVudDg2ODgzOTk5MA== max-sixty 5635139 2021-06-25T21:21:55Z 2021-06-25T21:21:55Z MEMBER

Yes very much so @Illviljan . But weirdly the linked PR is attempting to do that — so maybe this code path doesn't hit that change?

Spyder's profiler looks good! 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Very poor html repr performance on large multi-indexes 929818771
868767399 https://github.com/pydata/xarray/issues/5529#issuecomment-868767399 https://api.github.com/repos/pydata/xarray/issues/5529 MDEyOklzc3VlQ29tbWVudDg2ODc2NzM5OQ== Illviljan 14371165 2021-06-25T18:52:37Z 2021-06-25T18:52:37Z MEMBER

One way of solving it could be to slice the arrays to a smaller size but still showing the same repr. Because coords[0:12] seems easy to print, not sure how tricky it is to slice it in this way though.

I'm using https://github.com/spyder-ide/spyder for the profiling and general hacking.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Very poor html repr performance on large multi-indexes 929818771
868738004 https://github.com/pydata/xarray/issues/5529#issuecomment-868738004 https://api.github.com/repos/pydata/xarray/issues/5529 MDEyOklzc3VlQ29tbWVudDg2ODczODAwNA== max-sixty 5635139 2021-06-25T17:58:11Z 2021-06-25T17:58:11Z MEMBER

Yes, I think it's materializing the multiindex as an array of tuples. Which we definitely shouldn't be doing for reprs.

@Illviljan nice profiling view! What is that?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Very poor html repr performance on large multi-indexes 929818771
868735859 https://github.com/pydata/xarray/issues/5529#issuecomment-868735859 https://api.github.com/repos/pydata/xarray/issues/5529 MDEyOklzc3VlQ29tbWVudDg2ODczNTg1OQ== Illviljan 14371165 2021-06-25T17:54:00Z 2021-06-25T17:54:00Z MEMBER

I think it's some lazy calculation that kicks in. Because I can reproduce using np.asarray.

```python import numpy as np import xarray as xr

ds = xr.tutorial.load_dataset("air_temperature") da = ds["air"].stack(z=[...])

coord = da.z.variable.to_index_variable()

This is very slow:

a = np.asarray(coord)

da.repr_html() ``` 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Very poor html repr performance on large multi-indexes 929818771

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);