issue_comments

Table actions

5 rows where author_association = "MEMBER", issue = 221858543 and user = 1217238 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

user 1

  • shoyer · 5 ✖

issue 1

  • Sparse arrays · 5 ✖

author_association 1

  • MEMBER · 5 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
520675205 https://github.com/pydata/xarray/issues/1375#issuecomment-520675205 https://api.github.com/repos/pydata/xarray/issues/1375 MDEyOklzc3VlQ29tbWVudDUyMDY3NTIwNQ== shoyer 1217238 2019-08-13T03:31:14Z 2019-08-13T03:31:14Z MEMBER

This is working now on the master branch!

Once we get a few more kinks worked out, it will be in the next release.

I've started another issue for discussing how xarray could integrate sparse arrays better into its API: https://github.com/pydata/xarray/issues/3213

 
{
    "total_count": 4,
    "+1": 4,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Sparse arrays 221858543
504777412 https://github.com/pydata/xarray/issues/1375#issuecomment-504777412 https://api.github.com/repos/pydata/xarray/issues/1375 MDEyOklzc3VlQ29tbWVudDUwNDc3NzQxMg== shoyer 1217238 2019-06-23T18:54:33Z 2019-06-23T18:54:33Z MEMBER

It will need some experimentation, but I think things should be pretty close after NumPy 1.17 is released. Potentially it could be as easy as adjusting the rules xarray uses for casting in xarray.core.variable.as_compatible_data.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Sparse arrays 221858543
403155235 https://github.com/pydata/xarray/issues/1375#issuecomment-403155235 https://api.github.com/repos/pydata/xarray/issues/1375 MDEyOklzc3VlQ29tbWVudDQwMzE1NTIzNQ== shoyer 1217238 2018-07-06T21:49:27Z 2018-07-06T21:49:27Z MEMBER

Would it be an option to use dask's sparse support? http://dask.pydata.org/en/latest/array-sparse.html This way xarray could let dask do the main work.

In principle this would work, though I would prefer to support it directly in xarray, too.

I know that NetCDF4 has some conventions how to store sparse data, but do we have to implement our own conversion mechanisms for each sparse type?

Yes, we would need to implement a convention for handling sparse array data.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Sparse arrays 221858543
395223735 https://github.com/pydata/xarray/issues/1375#issuecomment-395223735 https://api.github.com/repos/pydata/xarray/issues/1375 MDEyOklzc3VlQ29tbWVudDM5NTIyMzczNQ== shoyer 1217238 2018-06-06T21:43:40Z 2018-06-06T21:43:40Z MEMBER

See also: https://github.com/pydata/xarray/issues/1938

The major challenge now is the dispatching mechanism, which hopefully http://www.numpy.org/neps/nep-0018-array-function-protocol.html will solve.

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Sparse arrays 221858543
294250748 https://github.com/pydata/xarray/issues/1375#issuecomment-294250748 https://api.github.com/repos/pydata/xarray/issues/1375 MDEyOklzc3VlQ29tbWVudDI5NDI1MDc0OA== shoyer 1217238 2017-04-14T22:46:10Z 2017-04-14T22:47:01Z MEMBER

Yes, I would say this is in scope, as long as we can keep most of the data-type specific logic out of xarray's core (which seems doable).

Currently, we define most of our operations on duck arrays in https://github.com/pydata/xarray/blob/master/xarray/core/duck_array_ops.py

There are a few other hacks throughout the codebase, which can find by searching for "dask_array_type": https://github.com/pydata/xarray/search?p=1&q=dask_array_type&type=&utf8=%E2%9C%93

It's pretty crude, but basically this would need to be extended to implement many of these methods on for sparse arrays, too. Ideally we would define xarray's adapter logic into more cleanly separated submodules, perhaps using multiple dispatch. Even better, we would make this public API, so you can write something like xarray.register_data_type(MySparseArray) to register a type as valid for xarray's .data attribute.

It looks like __array_ufunc__ will actually finally land in NumPy 1.13, which might make this easier.

See also https://github.com/pydata/xarray/pull/1118

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Sparse arrays 221858543

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);