issue_comments

Table actions

6 rows where issue = 166287789 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 2

issue 1

  • Pickle and .value vs. dask backend · 6 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
249436763 https://github.com/pydata/xarray/issues/902#issuecomment-249436763 https://api.github.com/repos/pydata/xarray/issues/902 MDEyOklzc3VlQ29tbWVudDI0OTQzNjc2Mw== crusaderky 6213168 2016-09-25T18:15:49Z 2016-09-25T18:15:49Z MEMBER

I'm done... I think. The result is less clean than I would have hoped - suggestions are welcome. https://github.com/pydata/xarray/pull/1018

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Pickle and .value vs. dask backend 166287789
249435675 https://github.com/pydata/xarray/issues/902#issuecomment-249435675 https://api.github.com/repos/pydata/xarray/issues/902 MDEyOklzc3VlQ29tbWVudDI0OTQzNTY3NQ== shoyer 1217238 2016-09-25T17:54:43Z 2016-09-25T17:54:43Z MEMBER

@crusaderky Let's just disable caching for dask.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Pickle and .value vs. dask backend 166287789
249417217 https://github.com/pydata/xarray/issues/902#issuecomment-249417217 https://api.github.com/repos/pydata/xarray/issues/902 MDEyOklzc3VlQ29tbWVudDI0OTQxNzIxNw== crusaderky 6213168 2016-09-25T11:47:31Z 2016-09-25T11:47:31Z MEMBER

Working on it now. What I didn't understand is if you want to disable caching for all backends (NetCDF etc.) or only for dask? The change for dask only is very straightforward. For all backends much less so...

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Pickle and .value vs. dask backend 166287789
239937280 https://github.com/pydata/xarray/issues/902#issuecomment-239937280 https://api.github.com/repos/pydata/xarray/issues/902 MDEyOklzc3VlQ29tbWVudDIzOTkzNzI4MA== shoyer 1217238 2016-08-15T21:38:16Z 2016-08-15T21:38:16Z MEMBER

This is where you can find the core caching logic on Variable objects: https://github.com/pydata/xarray/blob/56abba54ca4e89af570fd9cacc8f3ffcf5a5c4c7/xarray/core/variable.py#L257-L305

Here's where we define load on Dataset and DataArray: https://github.com/pydata/xarray/blob/56abba54ca4e89af570fd9cacc8f3ffcf5a5c4c7/xarray/core/dataset.py#L305-L327 https://github.com/pydata/xarray/blob/56abba54ca4e89af570fd9cacc8f3ffcf5a5c4c7/xarray/core/dataarray.py#L523-L536

As I mentioned before, let's add .compute() to evaluate and return a new object, and use it for .values instead of caching. .load() can remain unchanged for when users actually want to cache data. And we can definitely disable automatically loading data in pickles.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Pickle and .value vs. dask backend 166287789
239924378 https://github.com/pydata/xarray/issues/902#issuecomment-239924378 https://api.github.com/repos/pydata/xarray/issues/902 MDEyOklzc3VlQ29tbWVudDIzOTkyNDM3OA== crusaderky 6213168 2016-08-15T20:47:56Z 2016-08-15T20:47:56Z MEMBER

I'm happy to look into this - could you point me in the right direction?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Pickle and .value vs. dask backend 166287789
233692572 https://github.com/pydata/xarray/issues/902#issuecomment-233692572 https://api.github.com/repos/pydata/xarray/issues/902 MDEyOklzc3VlQ29tbWVudDIzMzY5MjU3Mg== shoyer 1217238 2016-07-19T16:40:39Z 2016-07-19T16:40:39Z MEMBER

I agree about loading data into memory automatically -- this behavior made sense before we used dask in xarray, but now it doesn't really.

We actually already have a .load() method for explicitly loading data into memory, though it might make sense to add .compute() as an alias, possibly without modifying the original dataset inplace.

I'm a little less certain about how to handle pickling data, because anytime you open a file from disk using open_dataset it's not going to pickle. But on the other hand, it's also not hard to explicitly write .load() or .compute() before using pickle or invoking multiprocessing.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Pickle and .value vs. dask backend 166287789

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);