html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/pull/947#issuecomment-246896876,https://api.github.com/repos/pydata/xarray/issues/947,246896876,MDEyOklzc3VlQ29tbWVudDI0Njg5Njg3Ng==,1217238,2016-09-14T03:35:04Z,2016-09-14T03:35:04Z,MEMBER,"OK, in it goes. Big thanks to @benbovy !
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,169588316
https://github.com/pydata/xarray/pull/947#issuecomment-246678848,https://api.github.com/repos/pydata/xarray/issues/947,246678848,MDEyOklzc3VlQ29tbWVudDI0NjY3ODg0OA==,4160723,2016-09-13T13:21:53Z,2016-09-13T13:21:53Z,MEMBER,"> I think the main (only?) thing left to do here is to remove the name argument you added to IndexVariable
Just removed it.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,169588316
https://github.com/pydata/xarray/pull/947#issuecomment-246560708,https://api.github.com/repos/pydata/xarray/issues/947,246560708,MDEyOklzc3VlQ29tbWVudDI0NjU2MDcwOA==,1217238,2016-09-13T03:08:37Z,2016-09-13T03:08:37Z,MEMBER,"I think the main (only?) thing left to do here is to remove the `name` argument you added to `IndexVariable`. For now, we can live with some `IndexVariable` objects with an inconsistent `name` (I'll remove the attribute shortly).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,169588316
https://github.com/pydata/xarray/pull/947#issuecomment-244539699,https://api.github.com/repos/pydata/xarray/issues/947,244539699,MDEyOklzc3VlQ29tbWVudDI0NDUzOTY5OQ==,4160723,2016-09-03T10:44:58Z,2016-09-03T10:44:58Z,MEMBER,"I've made changes according to your comments.
> I don't think it would be good idea to return a dict here because computing level values can be expensive and the levels aren't always necessary (they're stored internally as an Index for each level and integer codes).
I just thought that returning a dict could be a little more convenient (i.e., using a single call) if we need to get either one particular or several or all level(s) as `IndexVariable` object(s). However, I admit that this is certainly overkill and that it is actually not related to removing the `name` attr.
> BTW, I will be away over the holiday weekend (in the US), but I expect we will probably be able to merge this shortly after I get back.
Happy holidays! (I'll be on holiday next week too).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,169588316
https://github.com/pydata/xarray/pull/947#issuecomment-244518196,https://api.github.com/repos/pydata/xarray/issues/947,244518196,MDEyOklzc3VlQ29tbWVudDI0NDUxODE5Ng==,1217238,2016-09-03T01:14:48Z,2016-09-03T01:14:48Z,MEMBER,"> If we remove IndexVariable.name, then we can modify IndexVariable.get_level_variable such that it accepts one or more level names and returns an OrderedDict of {level_name: IndexVariable} items. That would be even better than the current implementation. Maybe we should then rename get_level_variable to get_levels_as_variables?
I think what `get_level_variable` returns is an independent matter? I don't think it would be good idea to return a dict here because computing level values can be expensive and the levels aren't always necessary (they're stored internally as an Index for each level and integer codes).
BTW, I will be away over the holiday weekend (in the US), but I expect we will probably be able to merge this shortly after I get back.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,169588316
https://github.com/pydata/xarray/pull/947#issuecomment-244514745,https://api.github.com/repos/pydata/xarray/issues/947,244514745,MDEyOklzc3VlQ29tbWVudDI0NDUxNDc0NQ==,4160723,2016-09-03T00:21:06Z,2016-09-03T00:21:06Z,MEMBER,"> Rather than adding an independent name attribute to IndexVariable, let's deprecate/remove IndexVariable.name instead (I can do this in a follow-up PR). Clearly it's useful to be able to use IndexVariable objects even for objects that do not correspond to ticks along a dimension, but for such objects, name is misleading. IndexVariable.name was never more than a convenient shortcut, and it's outlived it's usefulness.
I agree that `name` is misleading here. If we remove `IndexVariable.name`, then we can modify `IndexVariable.get_level_variable` such that it accepts one or more level names and returns an `OrderedDict` of {level_name: IndexVariable} items. That would be even better than the current implementation. Maybe we should then rename `get_level_variable` to `get_levels_as_variables`?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,169588316
https://github.com/pydata/xarray/pull/947#issuecomment-244512583,https://api.github.com/repos/pydata/xarray/issues/947,244512583,MDEyOklzc3VlQ29tbWVudDI0NDUxMjU4Mw==,1217238,2016-09-02T23:57:26Z,2016-09-02T23:57:26Z,MEMBER,"Rather than adding an independent `name` attribute to `IndexVariable`, let's deprecate/remove `IndexVariable.name` instead (I can do this in a follow-up PR). Clearly it's useful to be able to use `IndexVariable` objects even for objects that do not correspond to ticks along a dimension, but for such objects, `name` is misleading. `IndexVariable.name` was never more than a convenient shortcut, and it's outlived it's usefulness.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,169588316
https://github.com/pydata/xarray/pull/947#issuecomment-244507224,https://api.github.com/repos/pydata/xarray/issues/947,244507224,MDEyOklzc3VlQ29tbWVudDI0NDUwNzIyNA==,4160723,2016-09-02T23:07:14Z,2016-09-02T23:07:14Z,MEMBER,"@shoyer this is ready for another round of review. I don't see any remaining issue, I added some tests and I updated the docs.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,169588316
https://github.com/pydata/xarray/pull/947#issuecomment-244142205,https://api.github.com/repos/pydata/xarray/issues/947,244142205,MDEyOklzc3VlQ29tbWVudDI0NDE0MjIwNQ==,1217238,2016-09-01T16:55:12Z,2016-09-01T16:55:12Z,MEMBER,"> Should I write new tests specific to multi-index or should I modify existing tests (e.g., TestDataset.test_modify_inplace, TestDataset.tes_coord_properties, TestDataset.test_coords_modify, etc.) to include the multi-index case?
I would usually lean towards dedicated tests (e.g., `test_modify_multiindex_inplace`) but if it's easier to modify the original tests in a small way (e.g., for repr) feel free to take that approach
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,169588316
https://github.com/pydata/xarray/pull/947#issuecomment-244087601,https://api.github.com/repos/pydata/xarray/issues/947,244087601,MDEyOklzc3VlQ29tbWVudDI0NDA4NzYwMQ==,4160723,2016-09-01T14:00:02Z,2016-09-01T14:02:14Z,MEMBER,"Not sure how to write the tests for this PR, as there are quite many small changes spread in the API (e.g., repr, data object and coordinate properties, etc.). Should I write new tests specific to multi-index or should I modify existing tests (e.g., `TestDataset.test_modify_inplace`, `TestDataset.tes_coord_properties`, `TestDataset.test_coords_modify`, etc.) to include the multi-index case?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,169588316
https://github.com/pydata/xarray/pull/947#issuecomment-239159377,https://api.github.com/repos/pydata/xarray/issues/947,239159377,MDEyOklzc3VlQ29tbWVudDIzOTE1OTM3Nw==,4160723,2016-08-11T13:23:52Z,2016-08-11T15:33:50Z,MEMBER,"I just made some updates.
> I would add an attribute level_names to Coordinate
Done.
> It's much cheaper to call .get_level_values() after slicing a MultiIndex than before
Done, although it currently slices an arbitrary number (30) of first elements rather than calculating the number of elements needed for display.
> It seems like a fine rule to say that you cannot call .sel on both a level and the dimension name at the same time
Done.
> check uniqueness of level names at Dataset/DataArray creation (which is a good idea!)
I tried but it broke some existing tests. It actually triggered data loading for Coordinate objects (via calls to `to_index()`). I need to further investigate this.
> there is not much need for level indexing with a dictionary at all
Right, but in the current implementation this is still used internally.
> I would suggest putting the logic to create new variables for levels in the private _get_virtual_variable in dataset.py. [...] If possible, it would be nice if ds['time.day'] works even if time is a multi-index level.
Done. It should also work with multi-index levels although not tested yet.
> I'm currently leaning toward Option 3A...
I've chosen option 3A for the repr, but I can change it depending on others' opinions.
> What happens when you write ds.coords['level_1'] = ...? [...] probably better to raise an error and note that the MultiIndex should be modified instead.
Done.
> Should levels appear in ds.keys() or ds.coords.keys()?
They don't appear in there. If we keep Option 3A for the repr, I also think that we can avoid changes here.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,169588316
https://github.com/pydata/xarray/pull/947#issuecomment-238709101,https://api.github.com/repos/pydata/xarray/issues/947,238709101,MDEyOklzc3VlQ29tbWVudDIzODcwOTEwMQ==,1217238,2016-08-09T22:14:49Z,2016-08-09T22:14:49Z,MEMBER,"> As a recent xarray user, I indeed remember that I initially found confusing to have Dataset or DataArray ""coordinates"" that can be either Coordinate or Variable objects. I find that the name IndexVariable is more representative of the object.
Sounds good, I will do this in a separate PR.
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,169588316
https://github.com/pydata/xarray/pull/947#issuecomment-238549129,https://api.github.com/repos/pydata/xarray/issues/947,238549129,MDEyOklzc3VlQ29tbWVudDIzODU0OTEyOQ==,4160723,2016-08-09T13:16:50Z,2016-08-09T13:16:50Z,MEMBER,"> It seems like a fine rule to say that you cannot call .sel on both a level and the dimension name at the same time
Agreed. I have always the tendency to want to make things as flexible as possible, but this is definitely not needed here.
> I'm currently leaning toward Option 3A, but I don't have a strong opinion.
I'm also +1 for Option 3A. Maybe one little argument in favor of Options 3E or 3F is that they still show a consistent repr when a scalar is returned from the multi-index (see below), even though I don't like how they would display duplicate information.
```
>>> da.sel(level_1='a', level_2=0)
array([ 0.66068869, 0.20374398, 0.43967166, 0.09447679])
Coordinates:
x object ('a', 0)
* y (y) int64 0 1 2 3
```
> Possibly it would be a good idea to rename Coordinate -> IndexVariable
As a recent xarray user, I indeed remember that I initially found confusing to have Dataset or DataArray ""coordinates"" that can be either `Coordinate` or `Variable` objects. I find that the name `IndexVariable` is more representative of the object.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,169588316
https://github.com/pydata/xarray/pull/947#issuecomment-238053642,https://api.github.com/repos/pydata/xarray/issues/947,238053642,MDEyOklzc3VlQ29tbWVudDIzODA1MzY0Mg==,1217238,2016-08-06T23:07:25Z,2016-08-06T23:07:25Z,MEMBER,"I'm conflicted about how to handle the repr. On the one hand, I like how `*` indicates indexable variables. On the other hand, it should indeed be clear that these are MultiIndex levels, not dimensions in their own right (especially if they don't appear in `ds.coords.keys()` and the like). So maybe something closer to what we had before would be better.
Let me try to sketch out some concrete proposals to encourage the peanut gallery to speak up:
Option 1: no special indicator for the MultiIndex:
```
Coordinates:
* level_1 (x) object 'a' 'a' 'b' 'b'
* level_2 (x) int64 0 1 0 1
* y (y) int64 0 1 2 3
```
Option 2: both MultiIndex and levels in repr:
```
Coordinates:
* x (x) MultiIndex
* level_1 (x) object 'a' 'a' 'b' 'b'
* level_2 (x) int64 0 1 0 1
* y (y) int64 0 1 2 3
```
Option 3: both MultiIndex and levels in repr, different symbol for levels:
```
Coordinates:
* x (x) MultiIndex
- level_1 (x) object 'a' 'a' 'b' 'b'
- level_2 (x) int64 0 1 0 1
* y (y) int64 0 1 2 3
```
Option 4: both MultiIndex and levels in repr, different symbol for levels, with indentation:
```
Coordinates:
* x (x) MultiIndex
- level_1 (x) object 'a' 'a' 'b' 'b'
- level_2 (x) int64 0 1 0 1
* y (y) int64 0 1 2 3
```
A separate question (if we pick one of options 2-4) is how to represent the `MultiIndex` dtype and values (everything after the dimension name):
Option A: `MultiIndex` (as shown above)
Option B: `MultiIndex[level_0, level_1]`
Option C: `object MultiIndex`
Option D: `object MultiIndex[level_0, level_1]`
Option E: `MultiIndex ('a', 0) ('a', 1) ('b', 0) ('b', 1)`
Option F: `object ('a', 0) ('a', 1) ('b', 0) ('b', 1)` (current repr)
The tradeoffs here are whether or not we include the exact dtype information (`object`), and how explicitly/redundantly we display the values.
I'm currently leaning toward Option 3A, but I don't have a strong opinion.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,169588316
https://github.com/pydata/xarray/pull/947#issuecomment-238053437,https://api.github.com/repos/pydata/xarray/issues/947,238053437,MDEyOklzc3VlQ29tbWVudDIzODA1MzQzNw==,1217238,2016-08-06T23:00:17Z,2016-08-06T23:02:10Z,MEMBER,"I would suggest putting the logic to create new variables for levels in the private `_get_virtual_variable` in `dataset.py`. We already call this function for creating variables on demand in operations like `ds['time.month']`, so it's already called in all the right places (even in `ds.coords`, and so on). We could simply extend it to also check for MultiIndex levels and build those variables on demand, too, but only when necessary. If possible, it would be nice if `ds['time.day']` works even if `time` is a multi-index level.
This could get us most of the way there, but there are still a few things to watch out for:
1. What happens when you write `ds.coords['level_1'] = ...`? With the current implementation, I think this would create a new variable `level_1`. In an ideal world, maybe this would replace the MultiIndex level? For now, it is probably better to raise an error and note that the MultiIndex should be modified instead.
2. Should levels appear in `ds.keys()` or `ds.coords.keys()`? If we're printing them in the repr as peers to dimension coordinates, then maybe the answer should be yes? It could be confusing to have both the redundant levels and the multi-index coordinate in there, though. So maybe it's simpler to avoid changes here.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,169588316
https://github.com/pydata/xarray/pull/947#issuecomment-238050009,https://api.github.com/repos/pydata/xarray/issues/947,238050009,MDEyOklzc3VlQ29tbWVudDIzODA1MDAwOQ==,1217238,2016-08-06T21:35:24Z,2016-08-06T21:35:24Z,MEMBER,"This is very exciting to see!
A few thoughts on implementation:
Instead of always creating a dictionary of level coordinates, I would add an attribute `level_names` to Coordinate, which would default to `None` and be set to a tuple of `MultiIndex.names` if a Coordinate is created from a MultiIndex. This would make make checking for multi-index levels very cheap, even if we do need to iterate through every coordinate to find them.
It's much cheaper to call `.get_level_values()` after slicing a MultiIndex than before, e.g.,
```
In [12]: idx = pd.MultiIndex.from_product([np.linspace(0, 1, num=500), np.arange(1000)])
In [13]: %timeit idx.get_level_values(0)[:10]
1000 loops, best of 3: 1.28 ms per loop
In [14]: %timeit idx[:10].get_level_values(0)
10000 loops, best of 3: 101 µs per loop
```
It's even more extreme for larger indexes. If possible, we should use something closer to this approach when formatting coordinates.
> However, it is not possible anymore to mix level indexers and non-dict dimension indexers, e.g., da.sel(x='a', level_2=1) doesn't work but da.sel(x={'level_1': 'a'}, level_2=1) does, which is already quite flexible though.
I would actually be happy to disallow both, which might be even easy. It seems like a fine rule to say that you cannot call `.sel` on both a level and the dimension name at the same time. Actually, if we check uniqueness of level names at Dataset/DataArray creation (which is a good idea!), there is not much need for level indexing with a dictionary at all.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,169588316
https://github.com/pydata/xarray/pull/947#issuecomment-237997724,https://api.github.com/repos/pydata/xarray/issues/947,237997724,MDEyOklzc3VlQ29tbWVudDIzNzk5NzcyNA==,4160723,2016-08-06T01:15:02Z,2016-08-06T01:15:02Z,MEMBER,"In the example above, `DataArray.level_1` now returns a `DataArray` object, although I haven't found another way than creating a new `coordinates.DataArrayLevelCoordinates` class for this.
I wasn't happy with the initial implementation of using levels in `.sel`, it really added too much complexity. I re-implemented it and now I find it much cleaner. However, it is not possible anymore to mix level indexers and non-dict dimension indexers, e.g., `da.sel(x='a', level_2=1)` doesn't work but `da.sel(x={'level_1': 'a'}, level_2=1)` does, which is already quite flexible though.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,169588316