home / github

Menu
  • GraphQL API
  • Search all tables

issue_comments

Table actions
  • GraphQL API for issue_comments

9 rows where issue = 282178751 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 4

  • jakirkham 3
  • mrocklin 2
  • shoyer 2
  • jhamman 2

author_association 2

  • MEMBER 6
  • NONE 3

issue 1

  • Add compute=False keywords to `to_foo` functions · 9 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
367806166 https://github.com/pydata/xarray/issues/1784#issuecomment-367806166 https://api.github.com/repos/pydata/xarray/issues/1784 MDEyOklzc3VlQ29tbWVudDM2NzgwNjE2Ng== shoyer 1217238 2018-02-22T20:08:46Z 2018-02-22T20:08:46Z MEMBER

sync=False is a private, undocumented API. I think it's only found on the internal to_netcdf function in xarray.backends.api, not the to_netcdf method.

Currently, to_netcdf() will either return: - None if a path or file-like object was passed to to_netcdf(). - bytes representing the file's data if no path or file was provided.

In either case I think it could make sense to optionally replace the return value with a dask delayed object, which in turn evaluates to either None or bytes when computed.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Add compute=False keywords to `to_foo` functions 282178751
367166682 https://github.com/pydata/xarray/issues/1784#issuecomment-367166682 https://api.github.com/repos/pydata/xarray/issues/1784 MDEyOklzc3VlQ29tbWVudDM2NzE2NjY4Mg== jhamman 2443309 2018-02-21T00:10:04Z 2018-02-21T00:10:04Z MEMBER

What does ds.to_netcdf(...) usually return?

If sync == False the store is returned, otherwise nothing is returned.

The term future, when used in a Dask context, generally refers to something that is off computing asynchronously somewhere, rather than a token that holds onto a yet-to-be-submitted lazy graph.

Thanks for the clarification. I wasn't aware of that distinction but it does make sense.

What is store in this case?

A store is AbstractWritableDataStore, basically a wrapper class to allow us to read/write to various fileformats with various APIs under a common interface. Notably, each store has a writer attribute with a sync method that calls dask.array.store.


Another way to do this would be to have user code interact with the sync method directly:

```Python store = ds.to_netcdf('file.nc', sync=False)

store.sync calls store.writer.sync() which calls dask.array.sync

delayed_things = store.sync(compute=False) ```

This has the advantage of keeping the to_netcdf method a bit cleaner but does expose the AbstractWritableDataStore to user code which is typically not a public API object.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Add compute=False keywords to `to_foo` functions 282178751
367164232 https://github.com/pydata/xarray/issues/1784#issuecomment-367164232 https://api.github.com/repos/pydata/xarray/issues/1784 MDEyOklzc3VlQ29tbWVudDM2NzE2NDIzMg== jakirkham 3019665 2018-02-20T23:58:47Z 2018-02-20T23:58:47Z NONE

What is store in this case? Sorry not very familiar with how xarray does things.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Add compute=False keywords to `to_foo` functions 282178751
367163589 https://github.com/pydata/xarray/issues/1784#issuecomment-367163589 https://api.github.com/repos/pydata/xarray/issues/1784 MDEyOklzc3VlQ29tbWVudDM2NzE2MzU4OQ== mrocklin 306380 2018-02-20T23:55:25Z 2018-02-20T23:55:25Z MEMBER

The term future, when used in a Dask context, generally refers to something that is off computing asynchronously somewhere, rather than a token that holds onto a yet-to-be-submitted lazy graph.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Add compute=False keywords to `to_foo` functions 282178751
367163456 https://github.com/pydata/xarray/issues/1784#issuecomment-367163456 https://api.github.com/repos/pydata/xarray/issues/1784 MDEyOklzc3VlQ29tbWVudDM2NzE2MzQ1Ng== mrocklin 306380 2018-02-20T23:54:46Z 2018-02-20T23:54:46Z MEMBER

What does ds.to_netcdf(...) usually return? Typically when we specify compute=False we usually return a dask-like thing, often a dask.delayed object that evaluates to what would have been computed normally.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Add compute=False keywords to `to_foo` functions 282178751
367162456 https://github.com/pydata/xarray/issues/1784#issuecomment-367162456 https://api.github.com/repos/pydata/xarray/issues/1784 MDEyOklzc3VlQ29tbWVudDM2NzE2MjQ1Ng== jhamman 2443309 2018-02-20T23:49:41Z 2018-02-20T23:49:41Z MEMBER

@shoyer - Do you have thoughts on how this feature would present to the user? In #1811, I have added the compute keyword argument to to_netcdf and to_zarr and put a futures attribute on each store. So the workflow there would be something like:

```Python store = ds.to_netcdf('file.nc', compute=False)

dask.compute(store.futures) ```

Before I spend too much time on #1811, I want to get some buy in on the API for this feature.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Add compute=False keywords to `to_foo` functions 282178751
352036122 https://github.com/pydata/xarray/issues/1784#issuecomment-352036122 https://api.github.com/repos/pydata/xarray/issues/1784 MDEyOklzc3VlQ29tbWVudDM1MjAzNjEyMg== jakirkham 3019665 2017-12-15T15:38:14Z 2017-12-15T15:38:14Z NONE

In case anyone is curious, PR ( https://github.com/dask/dask/pull/2980 ) contains this work. Feedback welcome.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Add compute=False keywords to `to_foo` functions 282178751
351837521 https://github.com/pydata/xarray/issues/1784#issuecomment-351837521 https://api.github.com/repos/pydata/xarray/issues/1784 MDEyOklzc3VlQ29tbWVudDM1MTgzNzUyMQ== jakirkham 3019665 2017-12-14T21:13:30Z 2017-12-14T21:13:30Z NONE

Just to give a brief synopsis of what we are working in Dask in case it is valuable for this or other contexts, have given an overview of the relevant work below.

With Matthew's help am trying to add a keep argument to da.store. By default keep=False, which is the current behavior of da.store. If keep=True however, it returns Dask Arrays that can lazily load data written by da.store. Thus allowing the stored result to be linked to later computations before it is fully written. The compute argument of da.store affects whether to submit the storage tasks immediately (adding Futures into the resultant Dask Array) or whether to hold off until a later computation step triggers it.

This sort of functionality could be useful for a variety of situations including the one Matthew has described above. Also this could be useful for viewing partially computed results before they are totally done. Another use case could be more rapid batching of computations with many intermediate values. There is also an opportunity to re-explore caching in this context; thus, revisiting an area that many people have previously shown interest in.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Add compute=False keywords to `to_foo` functions 282178751
351785902 https://github.com/pydata/xarray/issues/1784#issuecomment-351785902 https://api.github.com/repos/pydata/xarray/issues/1784 MDEyOklzc3VlQ29tbWVudDM1MTc4NTkwMg== shoyer 1217238 2017-12-14T17:48:48Z 2017-12-14T17:48:48Z MEMBER

Yes, this sounds like a natural improvement to me!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Add compute=False keywords to `to_foo` functions 282178751

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);
Powered by Datasette · Queries took 11.996ms · About: xarray-datasette