issue_comments

Table actions

4 rows where issue = 970245117 and user = 14371165 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

  • Illviljan · 4 ✖

issue 1

  • Allow in-memory arrays with open_mfdataset · 4 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1528693660 https://github.com/pydata/xarray/pull/5704#issuecomment-1528693660 https://api.github.com/repos/pydata/xarray/issues/5704 IC_kwDOAMm_X85bHgOc Illviljan 14371165 2023-04-29T06:56:37Z 2023-04-29T06:58:26Z MEMBER

Those issues indeed has to be fixed if opening files lazily is the only option for xarray.

But xarray could also accept that chunks=None will (for now) load all the files to memory. If that's ok we can merge this now I believe. I suspect there are a few in-memory users out there that could make use of this.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Allow in-memory arrays with open_mfdataset 970245117
903442749 https://github.com/pydata/xarray/pull/5704#issuecomment-903442749 https://api.github.com/repos/pydata/xarray/issues/5704 IC_kwDOAMm_X8412XE9 Illviljan 14371165 2021-08-23T04:55:06Z 2021-08-23T04:55:06Z MEMBER

That the arrays would be loaded into memory is what you would expect if a user insists on using chunks=None right?

I just changed the default value to {}. So now it will behave as it did previously but with the possibility to load into memory for whatever reason you might have with small files.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Allow in-memory arrays with open_mfdataset 970245117
903176445 https://github.com/pydata/xarray/pull/5704#issuecomment-903176445 https://api.github.com/repos/pydata/xarray/issues/5704 IC_kwDOAMm_X8411WD9 Illviljan 14371165 2021-08-21T21:06:38Z 2021-08-21T21:06:38Z MEMBER

One way of making this less controversial is to also change the default value of chunks from None to {} here https://github.com/pydata/xarray/blob/48a9dbe7d8dc2361bc985dd9fb1193a26135b310/xarray/backends/api.py#L696 Then the default settings will behave the same as before. Although it's still not consistent with xr.open_datasets default parameters which mfdataset is just a thin wrapper around.

It is indeed bad practice to use dicts as default value but not completely uncommon, see for example: https://github.com/pydata/xarray/blob/48a9dbe7d8dc2361bc985dd9fb1193a26135b310/xarray/core/dataset.py#L2111

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Allow in-memory arrays with open_mfdataset 970245117
898356703 https://github.com/pydata/xarray/pull/5704#issuecomment-898356703 https://api.github.com/repos/pydata/xarray/issues/5704 IC_kwDOAMm_X841i9Xf Illviljan 14371165 2021-08-13T10:23:23Z 2021-08-13T10:27:30Z MEMBER

A lot of failing tests but they seem to just assume that open_mfdataset always returns dask arrays by default. Probably as simple as adding chunks={} in all these tests, but this is quite a breaking change.

Do you know the reason why chunks=chunks or {} is used in open_mfdataset, @aurghs?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Allow in-memory arrays with open_mfdataset 970245117

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);