home / github

Menu
  • GraphQL API
  • Search all tables

issue_comments

Table actions
  • GraphQL API for issue_comments

8 rows where author_association = "NONE" and issue = 221858543 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 5

  • Hoeze 3
  • fjanoos 2
  • rgommers 1
  • olgabot 1
  • lbybee 1

issue 1

  • Sparse arrays · 8 ✖

author_association 1

  • NONE · 8 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
526356476 https://github.com/pydata/xarray/issues/1375#issuecomment-526356476 https://api.github.com/repos/pydata/xarray/issues/1375 MDEyOklzc3VlQ29tbWVudDUyNjM1NjQ3Ng== fjanoos 923438 2019-08-29T20:52:10Z 2019-08-29T20:52:10Z NONE

@shoyer Is there documentation for using sparse arrays ? Could you point me to some example code ?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Sparse arrays 221858543
513589352 https://github.com/pydata/xarray/issues/1375#issuecomment-513589352 https://api.github.com/repos/pydata/xarray/issues/1375 MDEyOklzc3VlQ29tbWVudDUxMzU4OTM1Mg== fjanoos 923438 2019-07-21T21:32:23Z 2019-07-21T21:32:23Z NONE

Wondering what the status on this is ? Is there a branch with this functionality implemented - would love to give it a spin !

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Sparse arrays 221858543
511121578 https://github.com/pydata/xarray/issues/1375#issuecomment-511121578 https://api.github.com/repos/pydata/xarray/issues/1375 MDEyOklzc3VlQ29tbWVudDUxMTEyMTU3OA== rgommers 98330 2019-07-13T13:18:34Z 2019-07-13T13:18:34Z NONE

I haven't talked to anyone at SciPy'19 yet who was interested in sparse arrays, but I'll keep an eye out today.

And yes, this is a fun issue to work on and would be really nice to have!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Sparse arrays 221858543
402699810 https://github.com/pydata/xarray/issues/1375#issuecomment-402699810 https://api.github.com/repos/pydata/xarray/issues/1375 MDEyOklzc3VlQ29tbWVudDQwMjY5OTgxMA== Hoeze 1200058 2018-07-05T12:02:30Z 2018-07-05T12:02:30Z NONE

How should these sparse arrays get stored in NetCDF4? I know that NetCDF4 has some conventions how to store sparse data, but do we have to implement our own conversion mechanisms for each sparse type?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Sparse arrays 221858543
402699290 https://github.com/pydata/xarray/issues/1375#issuecomment-402699290 https://api.github.com/repos/pydata/xarray/issues/1375 MDEyOklzc3VlQ29tbWVudDQwMjY5OTI5MA== Hoeze 1200058 2018-07-05T12:00:15Z 2018-07-05T12:00:15Z NONE

Would it be an option to use dask's sparse support? http://dask.pydata.org/en/latest/array-sparse.html This way xarray could let dask do the main work.

Currently I load everything into a dask array by hand and pass this dask array to xarray. This works pretty good.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Sparse arrays 221858543
395009307 https://github.com/pydata/xarray/issues/1375#issuecomment-395009307 https://api.github.com/repos/pydata/xarray/issues/1375 MDEyOklzc3VlQ29tbWVudDM5NTAwOTMwNw== Hoeze 1200058 2018-06-06T09:39:43Z 2018-06-06T09:41:28Z NONE

I'd know a project which could make perfect use of xarray, if it would support sparse tensors: https://github.com/theislab/anndata

Currently I have to work with both xarray and anndata to store counts in sparse arrays separate from other depending data which is a little bit annoying :)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Sparse arrays 221858543
355383374 https://github.com/pydata/xarray/issues/1375#issuecomment-355383374 https://api.github.com/repos/pydata/xarray/issues/1375 MDEyOklzc3VlQ29tbWVudDM1NTM4MzM3NA== lbybee 4998171 2018-01-04T19:59:28Z 2018-01-04T19:59:28Z NONE

I'm interested to see if there have been any developments on this. I currently have an application where I'm working with multiple dask arrays, some of which are sparse (text data). It'd be worth my time to move my project to xarray, so I'm be interested in contributing something here if there is a need.

{
    "total_count": 3,
    "+1": 3,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Sparse arrays 221858543
311118338 https://github.com/pydata/xarray/issues/1375#issuecomment-311118338 https://api.github.com/repos/pydata/xarray/issues/1375 MDEyOklzc3VlQ29tbWVudDMxMTExODMzOA== olgabot 806256 2017-06-26T16:55:08Z 2017-06-26T16:55:08Z NONE

In case you're still looking for an application, gene expression from single cells (see data/00_original/GSM162679$i_P14Retina_$j.digital_expression.txt.gz) is very sparse due to high gene dropout. The shape is expression.shape (49300, 24760) and it's mostly zeros or nans. A plain csv from this data was 2.5 gigs, which gzipped to 300 megs.

Here is an example of using xarray to combine these files but my kernel keeps dying when I do ds.to_netcdf() :(

Hope this is a good example for sparse arrays!

{
    "total_count": 3,
    "+1": 3,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Sparse arrays 221858543

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);
Powered by Datasette · Queries took 14.861ms · About: xarray-datasette