issue_comments

Table actions

3 rows where author_association = "MEMBER" and issue = 606165039 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

issue 1

  • Add hook to get progress of long-running operations · 3 ✖

author_association 1

  • MEMBER · 3 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
619440580 https://github.com/pydata/xarray/issues/4000#issuecomment-619440580 https://api.github.com/repos/pydata/xarray/issues/4000 MDEyOklzc3VlQ29tbWVudDYxOTQ0MDU4MA== jhamman 2443309 2020-04-25T21:04:50Z 2020-04-25T21:04:50Z MEMBER

I agree with @rabernat and @TomNicholas that the easiest/best way to get progress information for computation is to use Dask.

We may want to consider something like a progress metric for a few Xarray operations that we can't offload to third party libraries. I'm mostly thinking of xarray's combine/merge/concat functions as opportunities for Xarray to improve its user experience. Simply telling the user how a task like open_mfdataset is progressing could go a long way toward improving the user's toleration for slow/challenging dataset construction operations (e.g. #1385).

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Add hook to get progress of long-running operations 606165039
619003228 https://github.com/pydata/xarray/issues/4000#issuecomment-619003228 https://api.github.com/repos/pydata/xarray/issues/4000 MDEyOklzc3VlQ29tbWVudDYxOTAwMzIyOA== TomNicholas 35968931 2020-04-24T13:17:46Z 2020-04-24T16:59:25Z MEMBER

If your operation uses dask then you can use the dask progress bar: ```python from dask.diagnostics import ProgressBar

with ProgressBar(): result = analyse(ds) result.plot() It looks like [########################################] | 100% Completed | 34.5 s ```

This works for any operation which calls dask.compute.

(Also by the way @cwerner because .to_dataframe() is a classmethod, you didn't have to initialise the Dataset in your example, you can just do dsout = xr.Dataset.from_dataframe(data))

 
{
    "total_count": 3,
    "+1": 3,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Add hook to get progress of long-running operations 606165039
619009819 https://github.com/pydata/xarray/issues/4000#issuecomment-619009819 https://api.github.com/repos/pydata/xarray/issues/4000 MDEyOklzc3VlQ29tbWVudDYxOTAwOTgxOQ== rabernat 1197350 2020-04-24T13:28:20Z 2020-04-24T13:28:52Z MEMBER

Whenever working with big data in xarray, the recommendation is to use dask, as @TomNicholas says.

It would be great if we could go directly from a dask.dataframe to an xarray dataset with dask arrays. Unfortunately this is not supported (yet): https://github.com/dask/dask/issues/6058

I believe our dev time would best be spent working on that integration, rather than building progress bars directly into xarray.

also see https://github.com/pydata/xarray/issues/3929

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Add hook to get progress of long-running operations 606165039

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);