issue_comments

Table actions

5 rows where issue = 137920337 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

author_association 2

issue 1

  • Index/dimension order not preserved when going from and to DataFrame · 5 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
458319339 https://github.com/pydata/xarray/issues/780#issuecomment-458319339 https://api.github.com/repos/pydata/xarray/issues/780 MDEyOklzc3VlQ29tbWVudDQ1ODMxOTMzOQ== stale[bot] 26384082 2019-01-28T22:02:00Z 2019-01-28T22:02:00Z NONE

In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity If this issue remains relevant, please comment here; otherwise it will be marked as closed automatically

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Index/dimension order not preserved when going from and to DataFrame 137920337
192248130 https://github.com/pydata/xarray/issues/780#issuecomment-192248130 https://api.github.com/repos/pydata/xarray/issues/780 MDEyOklzc3VlQ29tbWVudDE5MjI0ODEzMA== samwisehawkins 4641789 2016-03-04T11:43:28Z 2016-03-04T11:43:28Z NONE

Thanks, will have a look. 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Index/dimension order not preserved when going from and to DataFrame 137920337
191545231 https://github.com/pydata/xarray/issues/780#issuecomment-191545231 https://api.github.com/repos/pydata/xarray/issues/780 MDEyOklzc3VlQ29tbWVudDE5MTU0NTIzMQ== shoyer 1217238 2016-03-03T02:17:36Z 2016-03-03T02:17:36Z MEMBER

So I'm actually not sure whether to call this a bug or a feature. But I can explain why it works this way and maybe we can come up with something better.

With DataArray.to_series(), we are indeed careful to output the hierarchical index in the same order as the array dimensions. So it works there.

But on a Dataset, we don't necessarily have a unique ordering for the dimensions, because in general (though somewhat rarely in practice) the ordering of dimensions can differ between variables. This is why Dataset.dims returns a SortedKeysDict -- to avoid any implicit state derived off the order in which dimensions were added.

When converting a DataFrame, we currently build the MultiIndex independently of the data variables, so somewhat logically we simply take dimensions in sorted order. It might make more sense, though, to instead order levels in order of appearance on Dataset (non-index?) variables. I do try to avoid making heuristic choices like this, though, which is why it didn't make it into xarray already.

This code is pretty self-contained if you want to experiment and/or put together a PR: https://github.com/pydata/xarray/blob/v0.7.1/xarray/core/dataset.py#L1858-L1872

Basically, you need to ensure that ordered_dims is an OrderedDict with keys in the order you want for the resulting DataFrame.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Index/dimension order not preserved when going from and to DataFrame 137920337
191341328 https://github.com/pydata/xarray/issues/780#issuecomment-191341328 https://api.github.com/repos/pydata/xarray/issues/780 MDEyOklzc3VlQ29tbWVudDE5MTM0MTMyOA== samwisehawkins 4641789 2016-03-02T17:33:32Z 2016-03-02T17:33:32Z NONE

Moreover, I've just realised that the resulting DataFrame has a different ordering of the rows. So as far as I can tell, in order to get back the same DataFrame in a round trip to and from a Dataset, you need to to reorder the levels of the column index, then resort the rows according to that column index.

I'm doing this round trip as I'm storing DataFrames as netcdf files, and want to check that I can get do the round trip properly.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Index/dimension order not preserved when going from and to DataFrame 137920337
191314070 https://github.com/pydata/xarray/issues/780#issuecomment-191314070 https://api.github.com/repos/pydata/xarray/issues/780 MDEyOklzc3VlQ29tbWVudDE5MTMxNDA3MA== samwisehawkins 4641789 2016-03-02T16:29:13Z 2016-03-02T16:29:13Z NONE

This can easily be corrected using reorder_levels, but would be nice if this was automatic.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Index/dimension order not preserved when going from and to DataFrame 137920337

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);