issue_comments

Table actions

3 rows where issue = 479942077 and user = 923438 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 1

  • fjanoos · 3 ✖

issue 1

  • How should xarray use/support sparse arrays? · 3 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
526747770 https://github.com/pydata/xarray/issues/3213#issuecomment-526747770 https://api.github.com/repos/pydata/xarray/issues/3213 MDEyOklzc3VlQ29tbWVudDUyNjc0Nzc3MA== fjanoos 923438 2019-08-30T20:57:54Z 2019-08-30T20:57:54Z NONE

Thanks.

That solved that error but introduced another one.

Specifically - this is my dataframe

and this is the error that I get with sparse=True

My numpy version is definitely about 1.16

I also set this os.environ["NUMPY_EXPERIMENTAL_ARRAY_FUNCTION"]='1' just in case

Furthermore, I don't get this error when I don't set sparse=True ( I just get OOM errors but that's another matter) ...

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   How should xarray use/support sparse arrays? 479942077
526733257 https://github.com/pydata/xarray/issues/3213#issuecomment-526733257 https://api.github.com/repos/pydata/xarray/issues/3213 MDEyOklzc3VlQ29tbWVudDUyNjczMzI1Nw== fjanoos 923438 2019-08-30T20:10:43Z 2019-08-30T20:10:43Z NONE

I cloned the master branch and installed it using 'python setup.py develop'.

When I try to use the sparse data loading functionality as per python oo = xa.Dataset.from_dataframe( my_df, sparse=True ) I get the following error:

```

ModuleNotFoundError Traceback (most recent call last) <ipython-input-9-fce0ca6bc4c2> in <module> ----> 1 oo = xa.Dataset.from_dataframe( poly_df.iloc[:10000], sparse=True )

/mnt/local/xarray/xarray/core/dataset.py in from_dataframe(cls, dataframe, sparse) 4040 4041 if sparse: -> 4042 obj._set_sparse_data_from_dataframe(dataframe, dims, shape) 4043 else: 4044 obj._set_numpy_data_from_dataframe(dataframe, dims, shape)

/mnt/local/xarray/xarray/core/dataset.py in _set_sparse_data_from_dataframe(self, dataframe, dims, shape) 3936 self, dataframe: pd.DataFrame, dims: tuple, shape: Tuple[int, ...] 3937 ) -> None: -> 3938 from sparse import COO 3939 3940 idx = dataframe.index

ModuleNotFoundError: No module named 'sparse'

``` Any suggestions on what I need to do ?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   How should xarray use/support sparse arrays? 479942077
526710709 https://github.com/pydata/xarray/issues/3213#issuecomment-526710709 https://api.github.com/repos/pydata/xarray/issues/3213 MDEyOklzc3VlQ29tbWVudDUyNjcxMDcwOQ== fjanoos 923438 2019-08-30T18:53:44Z 2019-08-30T18:53:44Z NONE

Would it be possible that pd.{Series, DataFrame}.to_xarray() automatically creates a sparse dataarray - or we have a flag in to_xarray which allows controlling for this. I have a very sparse dataframe and everytime I try to convert it to xarray I blow out my memory. Keeping it sparse but logically as a DataArray would be fantastic.

 
{
    "total_count": 2,
    "+1": 2,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   How should xarray use/support sparse arrays? 479942077

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);