sha,message,author_date,committer_date,raw_author,raw_committer,repo,author,committer f9f4c730254073f0f5a8fce65f4bbaa0eefec5fd,"Fix multiindex level serialization after reset_index (#8672) * fix serialize multi-index level coord after reset * add regression test * update what's new --------- Co-authored-by: Deepak Cherian ",2024-01-31T17:42:28Z,2024-01-31T17:42:28Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 e2b6f3468ef829b8a83637965d34a164bf3bca78,"dirty workaround for mypy 1.5 error (#8142) Co-authored-by: Maximilian Roos <5635139+max-sixty@users.noreply.github.com>",2023-09-07T08:21:11Z,2023-09-07T08:21:11Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 f13da94db8ab4b564938a5e67435ac709698f1c9,"fix doctests: pandas 2.1 MultiIndex repr with nan (#8141) Co-authored-by: Mathias Hauser ",2023-09-05T08:35:36Z,2023-09-05T08:35:36Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 71177d481eb0c3547cb850a4b3e866af6d4fded7,"Add `Coordinates.assign()` method (#8102) * add Coordinates.assign method * update what's new * test typing Co-authored-by: Illviljan <14371165+Illviljan@users.noreply.github.com> * add docstring examples * fix doctests * fix doctests (spaces) * doctests typo again * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix assign test after merging main --------- Co-authored-by: Illviljan <14371165+Illviljan@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>",2023-09-01T13:28:16Z,2023-09-01T13:28:16Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 0f9f790c7e887bbfd13f4026fd1d37e4cd599ff1,"Better default behavior of the Coordinates constructor (#8107) * ``Coordinates.__init__`` create default indexes ... for any input dimension coordinate, if ``indexes=None``. Also, if another ``Coordinates`` object is passed, extract its indexes and raise if ``indexes`` is not None (no align/merge supported here). * add docstring examples * fix doctests * fix tests * update what's new",2023-08-31T07:35:46Z,2023-08-31T07:35:46Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 b136fcb679e9e70fd44b60688d96e75d4e3f8dcb,"Fix merge with compat=minimal (coord names) (#8104) * fix coord names after merge / compat minimal * update what's new * add assert in len(data_vars)",2023-08-30T07:57:35Z,2023-08-30T07:57:35Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 e5a38f6837ae9b9aa28a4bd063620a1cd802e093,better error message set index from scalar coord (#8109),2023-08-30T07:13:15Z,2023-08-30T07:13:15Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 1fedfd86604f87538d1953b01d6990c2c89fcbf3,"Refactor update coordinates to better handle multi-coordinate indexes (#8094) * generic warning implicitly wrap a pd.MultiIndex * refactor update_coords (assign) Fix more cases with multi-coordinate indexes: - do not try to align existing indexed coordinates with the new coordinates that will fully replace them - raise early if the new coordinates would corrupt the existing indexed coordinates - isolate PandasMultiIndex special cases so that it will be easier to drop support for it later (and warn now about deprecation) * fix alignment of updated coordinates when DataArray objects are passed as new coordinate objects * refactor Dataset.assign Need to update (replace) coordinates and data variables separately to ensure it goes through all (indexed) coordinate update checks. * fix and update tests * nit * fix mypy? * update what's new * fix import error * fix performance regression * nit * use warning util func and improve messages",2023-08-29T14:23:28Z,2023-08-29T14:23:28Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 4441f9915fa978ad5b276096ab67ba49602a09d2,"Expose ""Coordinates"" as part of Xarray's public API (#7368) * add indexes argument to Dataset.__init__ * make indexes arg public for DataArray.__init__ * Indexes constructor updates - easily create an empty Indexes collection - check consistency between indexes and variables * use the generic Mapping[Any, Index] for indexes * add wrap_pandas_multiindex function * do not create default indexes when not desired * fix Dataset dimensions TODO: check indexes shapes / dims for DataArray * copy the coordinate variables of passed indexes * DataArray: check dimensions/shape of index coords * docstrings tweaks * more Indexes safety Since its constructor can now be used publicly. Copy input mappings and check the type of input indexes. * ensure input indexes are Xarray indexes * add .assign_indexes() method * add `IndexedCoordinates` subclass + add `IndexedCoordinates.from_pandas_multiindex` helper. * rollback/update Dataset and DataArray constructors Drop the `indexes` argument or keep it as private API. When a `Coordinates` object is passed as `coords` argument, extract both coordinate variables and indexes and add them to the new Dataset or DataArray. * update docstrings * fix Dataset creation internal error * add IndexedCoordinates.merge_coords * drop IndexedCoordinates and reuse Coordinates * update api docs * make Coordinates init args optional * docstrings updates * convert to base variable when no index is given * raise when an index is given with no variable * skip create default indexes... ... When a Coordinates object is given to the Dataset constructor * invariant checks: maybe skip IndexVariable checks ... when check_default_indexes is False. * add Coordinates tests * more Coordinates tests * add Dataset constructor tests with Coordinates * fix mypy * assign_coords: do not create default indexes... ... when passing a Coordinates object * support alignment of Coordinates * clean-up * fix failing test (dataarray coords not extracted) * fix tests: prevent index conflicts Do not extract multi-coordinate indexes from DataArray if they are overwritten or dropped (dimension coordinate). * add Coordinates.equals and Coordinates.identical * more tests, docstrings, docs * fix assert_* (Coordinates subclasses) * review copy * another few tests * fix mypy * update what's new * do not copy indexes May corrupt multi-coordinate indexes. * add Coordinates fastpath constructor * fix sphinx directive * re-add coord indexes in merge (dataset constructor) This re-enables the optimization in deep_align that skips alignment for any alignable (DataArray) in a dict that matches an index key. * create coords with default idx: try a cleaner impl Coordinate variables and indexes extracted from DataArrays should be merged more properly. * some useful comments for later * xr.merge: add support for Coordinates objects * allow skip align for object(s) in merge_core This fixes the decrease in performance observed in Dataset creation benchmarks. When creating a new Dataset, the variables and indexes in `Coordinates` should already be aligned together so it doesn't need to go through the complex alignment logic once again. `Coordinates` indexes are still used to align data variables. * fix mypy * what's new tweaks * align Coordinates callbacks: don't reindex data vars * fix Coordinates._overwrite_indexes callback mypy was rightfully complaining. This callback is called from Aligner only, which passes the first two arguments and ignores the rest. * remove merge_coords * futurewarning: pass multi-index via data vars * review comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix circulat imports * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * typing: add Alignable protocol class * try fixing mypy error (Self redefinition) * remove Coordinate alias of Variable Much water has flowed under the bridge since it has been renamed. * fix groupby test * doc: remove merge_coords in api reference * doc: improve docstrings and glossary * use Self type annotation in Coordinate class * better comment * fix Self undefined error with python < 3.11 Pyright displays an info message ""Self is not valid in this context"" but most important this should avoid runtime errors with python < 3.11. --------- Co-authored-by: Illviljan <14371165+Illviljan@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Deepak Cherian ",2023-07-21T20:40:03Z,2023-07-21T20:40:03Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 7234603781768728b3fd544cdcaca991466d4a44,"Add documentation on custom indexes (#6975) * improve Index base class type annotations Use T_Index generic when possible. * import Index base class in Xarray root namespace * import IndexSelResult into Xarray root namespace * wip: Index API docstrings * wip: doc: add how to add custom index section * add Index method docstrings * add user guide on how to create a custom index * review comments + tweaks * update what's new * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Apply uncontroversial suggestions from Deepak's code review Co-authored-by: Deepak Cherian * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Apply more suggestions from code review Co-authored-by: Deepak Cherian * Link to source code for PandasIndex and PandasMultiIndex --------- Co-authored-by: Thomas Nicholas Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Deepak Cherian ",2023-07-17T23:23:21Z,2023-07-17T23:23:21Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 d6d24507793af9bcaed79d7f8d3ac910e176f1ce,"Some alignment optimizations (#7382) * compare indexes: return early if all same objects This may happen in some (rare?) cases where the objects to align share the same indexes. * avoid re-indexing when not needed If all unindexed dimension sizes match the indexed dimension sizes in the objects to align, we don't need re-indexing. * add benchmark * update what's new Co-authored-by: Illviljan <14371165+Illviljan@users.noreply.github.com>",2023-01-05T21:25:55Z,2023-01-05T21:25:55Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 8938d390a969a94275a4d943033a85935acbce2b,"Fix assign_coords resetting all dimension coords to default index (#7347) * fix merge_coords and collect_variables_and_indexes * add test * update what's new * update what's new with pull-request link",2022-12-02T16:32:40Z,2022-12-02T16:32:40Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 f93b467db5e35ca94fefa518c32ee9bf93232475,"Fix to_index(): return multiindex level as single index (#7105) * to_index(): return multiindex level as single index * try fix broken tests Add Variable._to_index method used internally and returns multi-index for level variables. * fix groupby test Add DataArray._to_index * one small comment * update release notes * fix docs * refactor / move safe_cast_to_index",2022-10-12T14:12:47Z,2022-10-12T14:12:47Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 513ee34f16cc8f9250a72952e33bf9b4c95d33d1,Fix Dataset.assign_coords overwriting multi-index (#7101),2022-09-28T18:02:14Z,2022-09-28T18:02:14Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 e678a1d7884a3c24dba22d41b2eef5d7fe5258e7,"Add set_xindex and drop_indexes methods (#6971) * temporary API to set custom indexes * add the temporary index API to DataArray * add options argument to Index.from_variables() It allows passing options to the constructor of a custom index class (if any). The **options arguments of Dataset.set_xindex() are passed through. Also add type annotations to set_xindex(). * fix mypy * remove temporary API warning * add the Index class in Xarray's root namespace * improve set_xindex docstrings and add to api.rst * remove temp comments * special case for pandas multi-index dim coord * add tests for set_xindex * error message tweaks * set_xindex with 1 coord: avoid reodering coords * mypy fixes * add Dataset and DataArray drop_indexes methods * improve assert_no_index_corrupted error msg * drop_indexes: add tests * add drop_indexes to api.rst * improve docstrings of legacy methods * add what's new entry * try using correct typing w/o mypy complaining * make index_cls arg optional Try setting a pandas (multi-)index by default. * docstrings fixes and tweaks * make Index.from_variables options arg keyword only * improve set_xindex invalid coordinates error msg * add xarray.indexes namespace * type tweaks Co-authored-by: Keewis ",2022-09-28T07:25:15Z,2022-09-28T07:25:15Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 a042ae69c0444912f94bb4f29c93fa05046893ed,"Review (re)set_index (#6992) * review reset_index + tests Restore old behavior, i.e., - drop the multi-index dimension name (even if drop=False) unless reset_index still returns a multi-index - rename the level coordinate to the dimension name if the multi-index is reduced to a single index - drop the whole multi-index if its dimension coordinate is given as argument Fix IndexVariable -> Variable conversion * reset_index: fix dropped dimension(s) * reset_index: fix other tests * review set_index - Convert the coordinates left unindexed from IndexVariable to Variable - Keep multi-index coordinates next to each other in a consistent order * set_index with single index: preserve coord order * update what's new",2022-09-27T10:35:37Z,2022-09-27T10:35:37Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 45c0a114e2b7b27b83c9618bc05b36afac82183c,"Raise UserWarning when rename creates a new dimension coord (#6999) * warn when rename creates a new dimension coord UseWarning: no index is created anymore. * update what's new",2022-09-27T09:33:40Z,2022-09-27T09:33:40Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 9d1499e22e2748eeaf088e6a2abc5c34053bf37c,misc. fixes for Indexes with pd.Index objects (#7003),2022-09-23T07:30:37Z,2022-09-23T07:30:37Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 4880012ddee9e43e3e18e95551876e9c182feafb,"Fix aligned index variable metadata side effect (#6857) * alignment: fix index variable metadata side effect * add regression test * assert no side effect on original objects * update what's new Co-authored-by: Deepak Cherian ",2022-08-31T07:16:14Z,2022-08-31T07:16:14Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 facafac359c39c3e940391a3829869b4a3df5d70,Fix concat with scalar coordinate (wrong index type) (#6443),2022-04-06T01:19:47Z,2022-04-06T01:19:47Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 d8fc34660f409d4c6a7ce9fe126d126e4f76c7fd,"Speed-up multi-index html repr + add display_values_threshold option (#6400) * add _repr_html_ for PandasMultiIndexingAdapter This may greatly speed-up the html repr of Xarray objects with multi-indexes This optimized _repr_html_ is now used for formatting the array detailed view of multi-index coordinates, instead of converting the full index / levels to numpy arrays before formatting it. * update release notes * nit * add display_values_threshold * fix last merge main",2022-03-29T07:05:31Z,2022-03-29T07:05:31Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 009b15461bf1ad4567e57742e44db4efa4e44cc7,Fix concat scalar coord dtype (#6418),2022-03-28T16:05:00Z,2022-03-28T16:05:00Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 321c5608a3be3cd4b6a4de3b658d1e2d164c0409,fix DataArray groupby returning a Dataset (#6394),2022-03-21T15:26:20Z,2022-03-21T15:26:20Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 c604ee1fe852d51560100df6af79b4c28660f6b5,reindex: fix missing variable metadata (#6389),2022-03-21T07:53:04Z,2022-03-21T07:53:04Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 83f238a05a82fc85dcd7346f758ba3bea0416181,Fix concat with scalar coordinate (#6385),2022-03-21T04:49:22Z,2022-03-21T04:49:22Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 067b2e86e6311e9c37e0def0c83cdb9a1a367a74,isel: convert IndexVariable to Variable if index is dropped (#6388),2022-03-21T04:47:47Z,2022-03-21T04:47:47Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 fed852073eee883c0ed1e13e28e508ff0cf9d5c1,"fix dataset groupby combine dataarray func (#6386) Dataset._overwrite_indexes and DataArray._overwrite_indexes have different parameter names (for consistency with other internals).",2022-03-20T18:55:25Z,2022-03-20T18:55:25Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 03b6ba1e779b0d1829ca7b2e8f5da4d9c39ece6f,"fix concat with variable or dataarray as dim (#6387) Propagate attrs.",2022-03-20T18:53:46Z,2022-03-20T18:53:46Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 3ead17ea9e99283e2511b65b9d864d1c7b10b3c4,"Explicit indexes (#5692) * no need to wrap pandas index in lazy index adapter * multi-index default level names * refactor setting Dataset/DataArray default indexes * update multi-index (text) repr Notes: - move the multi-index formatting logic into PandasMultiIndexingAdapter._repr_inline_ - inline repr: check for _repr_inline_ implementation first * remove print * minor fixes and improvements * fix dtype of index variables created from Index * fix multi-index selection regression See https://github.com/pydata/xarray/issues/5691 * check conflicting multi-index level names * update formatting (text and html) * check level name conflicts for midx given as coord * intended behavior or unwanted side effect? see #5732 * get rid of multi-index virtual coordinates Not totally yet: need to refactor set_index / reset_index * add level coords in indexes & keep coord order * fix copying multi-index level variable data * collect index for multi-index level variables Avoid re-creating the indexes for dimension variables. Collect then directly instead. Note: the change here is working for building new Datasets but I haven't checked other cases like merging different objects, etc. So I'm not sure this is the right approach. * wip refactor label based selection - Index.query must now return a mapping of {dim_name: positional_indexer} as indexes may be based on several coordinates with different dimensions - Added `group_coords_by_index` utility function (not used yet, not sure we'll need it) TODO: - Update DataArray selection - Update .loc and other places using remap_label_indexers - Fix selection of multi-index that returns only scalar coordinates * fix index query tests * fix multi-index adapter getitem scalar Multi-index level variables now return the scalar value that corresponds to the level instead of the multi-index tuple element (all levels). Also get rid of PandasMultiIndexingAdapter.__getitem__ cache optimization, which doesn't work with level scalar values and was premature optimization anyway. * wip refactor label based selection Fixed renamed dimension in the case of multi-index -> single index Updated DataArray._overwrite_indexes Dirty fix for alignment (not tested yet) * wip: deeper refactoring label-based sel Created QueryResult and MergedQueryResults classes for convenience. * fix some tests + minor tweaks * fix indexing PandasMultiIndexingAdapater When level is not None: - if result is another adapter: propagate it properly - if result is a numpy adapter: use level values * refactor cast label indexer to coord dtype Make the fix in #3153 specific to pandas indexes (i.e., do not apply it to other, custom indexes). See #5697 for details. This should also fix #5700 although no test has been added yet (we need to refactor set_index first). * better handling of multi-index level labels * label-based selection tweaks and fixes - Use a dataclass for QueryResult - Pass Variables and DataArrays indexers un-normalized to Index.query(). Indexes have the responsibility of returning the expected types for positional (dimension) indexers by adding back dimensions and coordinates if needed. - Typing fixes and tweaks * sel: propagate multi-index vars attrs/encoding Related to https://github.com/pydata/xarray/issues/1366 * wip refactor rename Still needs tests Also need to address the problem of multi-index level coordinates (data adapters currently not updated). We'll probably need for `Index.rename()` to also return new index variables? * wip refactor rename: return new vars from Index * refactor rename: update tests Still need to address default indexes internal checks (these are disabled for now). * typing tweaks * fix html formatting: dims w/ or w/o index * minor fixes and tweaks * wip refactor set_index * wip refactor set_index - Refactor reset_index - Improve creating new xarray indexes from pandas indexes with proper propagation of variable metadata (dtype, attrs, encoding) * refactor reorder_levels * Set pd.MultiIndex name from dim name Closes #4542 * .sel() with multi-index: return scalar coords ..instead of dropping those coordinates Closes #1408 once tests are fixed/updated * fix multi-index level coordinate inline repr * wip refactor stack Some changes: Multi-indexes are created only if the stacked dimensions each have a single coordinate index. In the other cases, multi-indexes are not created, which means that it is an irreversible operation (unstack will not work) Multi-indexes are created from the stacked coordinate variables and not anymore from the unstacked dimension indexes (level product). There's a significant decrease in performance but it is probably acceptable since it's now possible to avoid the creation of multi-indexes. It is possible to stack a dimension with a multi-index, but it drops the index. Otherwise it would make it hard for unstack() to figure out what's going on. It makes it clear that this is an irreversible operation. * stack: better rule for add/skip multi-index It is more robust and allows creating a multi-index from non-default (non-dimension) coordinates, as long as there's one and only one 1-d indexed coordinate for each dimension to stack. * add utility methods to class Indexes This removes some ugly & duplicate patterns. * re-arrange class Indexes internals * wip refactor stack / unstack stack: revert to old way of creating multi-index unstack: support non-default (non-dimension) multi-index (as long as there is exactly one multi-index per specified dimension) * add PandasMultiIndex.from_product_variables Refactored utils.multiindex_from_product_levels utility function into a new PandasMultiIndex classmethod. * unstack: propagate index coordinate metadata * wip: fix/update tests * fix/refactor reindex * functools.cached_property is since py38 Use the good old way as we still support py37 * update reset_coords * fix ipython key completion test make sure the stacked dataset used has a multi-index * update test_map_index_queries * do not coerce bool indexer as float coord dtype Fixes #5727 * add Index.create_variables() method This will probably be used instead of including index coordinate variables in the signature (return type) of many methods of ``Index``. It has several advantages: - More DRY, and for custom indexes that do not need to create coordinate variables with special data adapters, it's easier to just skip implementing this method and not bother with returning empty dicts in other methods - This allows to decouple index vs. coordinates creation. For many cases this can be done at the same time but for some cases like alignment this is useful - It's a more elegant solution when we need to propagate metadata (attrs, encoding) * improve Dataset/DataArray indexes proxy class One possible solution to the recurring problem of accessing coordinate variables related to a given index in some of Xarray's internals. I feel that the ``Indexes`` proxy is the right place for storing references to those indexed coordinate variables. It's safer than relying on a public attribute/property of ``Index``. * align Indexes API with Coordinates API Expose similar properties: - variables - dims * clean-up formatting using updated Indexes API * wip: refactor alignment * wip refactor alignment Almost done rewriting `align`. * tweaks and fixes - Some fixes and clean-up in new implementation of align - Add Index.join method (only for inner/outer join) - Tweak index generic types - IndexVars type: use Variable instead of IndexVariable (IndexVariable may eventually be dropped) * wip refactor alignment and reindex Imported all re-indexing logic into the `Alignator` class, which can be reused directly for both `align()` and `obj.reindex()`. TODO: - import override indexes into `Alignator` - connect Alignator to public API entry points - clean-up (remove old implementation functions) * wip refactor alignment / reindex: fixes and tweaks * wip refactor alignment (support join='override') * wip alignment: clean-up + tests I will fix mypy errors later. * refactor alignment fix tests & types annotations All `*align*` and `*reindex*` tests are passing! (locally) Mypy doesn't complain. TODO: refactor `interp` and `interp_like` (I only plan to fix it with the new Aligner class for now) * refactor interp and interp_like It now reuses the new alignment/reindex internals, but it still only supports dimension coordinates with a single index, * wip review merge * refactor swap_dims * refactor isel * add create_index option to stack * add Index.stack and Index.unstack methods Also added a `index_cls` parameter to `Dataset.stack` and `DataArray.stack`. Custom indexes may thus implement their own logic for stack/unstack, with the current limitation that a `pandas.MultiIndex` is still explicitly required for unstack optimized versions. * fix PandasIndex.isel with scalar indexers Fix concat tests and some groupby tests * fix index adapter (variable) dtype * more indexes invariants checks Check consistency between PandasIndex objects and coordinate variables data adapters (PandasIndexingAdapter). * fix DataArray.isel() * refactor Dataset/DataArray copy Filter indexes and preserve unique index objects for multi-coordinate indexes. Also fixed indexes tests (stack/unstack). * refactor computation (and merge) * minor fixes and tweaks * propagate_indexes -> filter_indexes_from_coords We can't just look at excluded dimensions anymore... * Update xarray/core/coordinates.py * refactor expand_dims * merge: avoid compare same indexes more than once * wip refactor update coords Check in merge that prioritized elements won't corrupt collected indexes. * refactor update/remove coords Fixes and tweaks Refactored assign, update, setitem, delitem, drop_vars Added or updated tests * misc. fixes * fix Dataset,__delitem__ * fix Dataset.reset_coords (default coords) * refactor Dataset.from_dataframe * Fix .sel with DataArray and multi-index * PandasIndex.from_variables: preserve wrapped index Prevent converting special index types like ``pd.CategoricalIndex`` when such objects are wrapped in xarray variables. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * two minor fixes after merging main * filter_indexes_from_coords: preserve index order * implicit multi-index from coord: fix edge case * groupby combine: fix missing index in result When coord is None * implicit multi-index coord: more robust fix Also cover the cases where indexes are equal but not identical (caused a couple of tests failing). Perfs may not be a concern for such edge case? * backward compat fix: multi-index given as data-var ... in Dataset constructor * refactor concat Summary: - Add the ``Index.concat()`` class method with implementations for ``PandasIndex`` and ``PandasMultiIndex`` (adapted from ``IndexVariable.concat()``). - Use ``Index.concat()`` to create new indexes (and their corresponding variables) in ``_dataset_concat``. Fallback to variable concatenation (with default index) when no implementation is given for ``Index.concat`` or when no all the variables have an index. - Refactored merging the other variables (also merge the indexes) in ``_dataset_concat``. This refactor is incomplete. It should work with Pandas (multi-)indexes but it is likely that it won't work with meta-indexes involving multiple dimensions. We probably need to update ``_calc_concat_over`` to take into account such meta-indexes and their related coordinates. Other limitation (?) we use here the index of the 1st dataset for the concatenation (i.e., ``Index.concat``). No specific check is made on the type and/or coordinates of the other indexes. * sel drop=True: remove multi-index scalar coords * PandasIndex.from_variables(multi-index level var) Make sure it gets the level index (not the whole multi-index). * reindex: disable invalid dimension check From the docstrings: mis-matched dimensions are simply ignored * add index concat tests + fix positions type * add Indexes.get_all_dims convenient method * refactor pad * unstack: return copies of mindex.levels Fix error when trying to set the name of the extracted level indexes. * strip_units: prevent index->array conversion which caused alignment conflicts due to multi-index variable converted to non-index variable. * attach_units: do not preserve coord multi-indexes To prevent conflicts when trying to implicitly create multi-index coordinates. * concat: avoid multiple loads of dask arrays * groupby mindex coord: propagate level names * Dataset.copy: don't reset indexes if data is given The `data` parameter accepts new data for data variables only, it won't affect the indexes. * reindex/align: fix coord dtype of new indexes Use `as_compatible_data` to get the right dtype to assign to the new index coordinates created from reindex indexers. * fix doctests * to_stacked_array: do not coerce level dtypes If the intent was to propagate the original coordinate dtypes, this is now handled by PandasMultiIndex.stack. This fixes the case where multi-index ``.levels`` do not return labels with ""nan"" values (if any) but where the coordinate does, which resulted in dtype mismatch between the coordinate variable dtype (e.g., coerced to int64) and the actual label values (float64 due to nan values), eventually causing numpy errors trying to coerce nan value to int64 when indexing the coordinate. * fix indent * doc: fix user-guide build errors * stack: fix new index variables not in coordinates * unstack full-reindex: fix alignment errors We need a mapping of all index coordinate names to the full index so that Aligner can find matching indexes. (Note: reindex may eventually be refactored to be a bit more clever so that providing only a `{dim: indexer}` will be enough in this case?) * PandasIndex coord dtype: avoid convert index->array * refactor diff * quick fix level coord dtype int32/64 on win * dask presist/compute dataarray: propagate indexes * refactor roll * rename Index.query -> Index.sel Also rename: - QueryResult -> IndexSelResult - merge_query_results -> merge_sel_results * remove Index.union and Index.intersection Those are not used elsewhere. ``Index.join`` is used instead for alignment. * use future annotations in indexes.py * Index.rename: return only the new index Create new coordinate variables using Index.create_variables instead (get rid of PandasIndex.from_pandas_index). * Index.stack: return only the new index Create new coordinate variables using Index.create_variables instead (get rid of PandasIndex.from_pandas_index) * PandasMultiIndex class methods: return only the index * wip get rid of Pandas(Multi)Index.from_pandas_index * remove Pandas(Multi)Index.from_pandas_index * Index.from_variables: return the new index only Use Index.create_variables to get the new coordinate variables. * rename: propagate_attrs_encoding not needed Index.create_variables already propagates coordinate variable metadata. * align exclude dims: pass through indexes * fix set_index append=True Level variables may have updated names in this case: single index + one level. * refactor default_indexes and re-enable invariant check Default indexes invariant check is now optional (enabled by default). * fix formatting errors * fix type * Dataset._indexes, DataArray._indexes: always dict Default indexes are created by the default constructors. Indexes are always passed explicitly by the fastpath constructors. * remove _level_coords property It was only used for plotting and has been replaced by simpler logic. * clean-up Use ``_indexes`` instead of ``.xindexes`` when possible, for speed. ``xindexes`` is not cached: it would return inconsistent results if updating the object in-place. Fix some types. * optimize Dataset/DataArray copy Avoid copying pandas indexes twice (indexes + coordinate variables) * add default implementation for Index.copy Better than raising an error? * add/fix indexes tests * tweak indexes formatting This will need more work (in a follow-up PR) to be consistent with the other data model components (i.e., coordinates and data variables): add summary (inline) and detailed reprs, maybe group by coordinates, etc. * fix doctests * PandasIndex.copy: avoid too many pd.Index copies * DataArray stack test: revert to original Now that default (range) indexes are created again (create_index=True). * test indexes/indexing: rename query -> sel * alignment: use future annotations * misc. tweaks * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Aligner tweaks Access aligned objects through the ``.result`` attribute. Enable type checking in internal methods. * remove assert_unique_multiindex_level_names It is not needed anymore as multi-indexes have their own coordinates and we now check for possible name conflicts when ""unpacking"" multi-index levels at Dataset / DataArray creation. * remove propagate_attrs_encoding Only used in one place. * assert -> check + ValueError * misc. fixes and tweaks * update what's new * concat: remove fall-backs to Variable.concat Raise an error instead when trying to concat a mix of indexed and un-indexed coordinates or when the index doesn't support concat. We still support the concatenation of a mix of scalar and dimension indexed coordinates by creating a PandasIndex on-the-fly for scalar coordinates. * fix flaky test Co-authored-by: Illviljan <14371165+Illviljan@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: keewis ",2022-03-17T17:11:40Z,2022-03-17T17:11:40Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 4bb9d9c6df77137f05e85c7cc6508fe7a93dc0e4,"Refactor index vs. coordinate variable(s) (#5636) * split index / coordinate variable(s) - Pass Variable objects to xarray.Index constructor - The index should create IndexVariable objects (`coords` attribute) - PandasIndex: IndexVariable wraps PandasIndexingAdpater wraps pd.Index * one PandasIndexingAdapter subclass for multiindex * fastpath Index init + from_pandas_index classmethods * use classmethod constructors instead * add Index.copy and Index.__getitem__ methods * wip: clean-up Revert some changes made in #5102 + additional (temporary) fixes. * clean-up * add PandasIndex and PandasMultiIndex tests * remove unused import * doc: update what's new * use xindexes in map_blocks + temp fix Dataset constructor doesn't accept xarray indexes yet. Create new coordinates from the underlying pandas indexes. * update what's new with #5670 * typo",2021-08-09T07:56:56Z,2021-08-09T07:56:56Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 9daf9b13648c9a02bddee3640b80fe95ea1fff61,"Internal refactor of label-based data selection (#5322) * xindexes also returns multi-index levels as keys * wip: move label selection into PandasIndex add Index.query() method split pd.Index vs. pd.MultiIndex logic * Revert ""xindexes also returns multi-index levels as keys"" This reverts commit 261fb7880523db79b5bc3715477374d089e9a574. Let's keep this for later. There are too many places in Xarray that assume that xindexes keys are dimension names. * fix broken tests * remove old code + move/update tests * remove duplicate function * add PandasMultiIndex class + refactor query impl * remove PandasIndex.from_variables for now Add it later in the refactoring when it will be needed elsewhere (e.g., in ``set_index``). * fix broken tests Is this what we want? * prevent loading values for xarray objs in slice * update what's new",2021-06-08T09:35:54Z,2021-06-08T09:35:54Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 2b38adc1bdd1dd97934fb061d174149c73066f19,"Cast PandasIndex to pd.(Multi)Index (#5385) * fastpath cast Xarray's PandasIndex to pd.Index Also make sure that a multi-index with one unique level are not cast to a simple pd.Index * update tests * [skip-ci] update what's new",2021-05-28T08:28:11Z,2021-05-28T08:28:11Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 6e14df62f0b01d8ca5b04bd0ed2b5ee45444265d,"Flexible indexes: add Index base class and xindexes properties (#5102) * add IndexAdapter class + move PandasIndexAdapter * wip: xarray_obj.indexes -> IndexAdapter objects * fix more broken tests * fix merge glitch * fix group bins tests * add xindexes property Use it internally instead of indexes * rename IndexAdapter -> Index * rename _to_index_adpater (typo) -> _to_xindex * add Index.to_pandas_index() method Also improve xarray_obj.indexes property implementation * rename PandasIndexAdpater -> PandasIndex * update index type in tests * ensure .indexes only returns pd.Index objects * PandasIndex: normalize other index in cmp funcs * fix merge lint errors * fix PandasIndex union/intersection * [skip-ci] add TODO comment about index sizes * address more PR comments * [skip-ci] update what's new * fix coord_names normalization * move what's new entry to unreleased section",2021-05-11T08:21:25Z,2021-05-11T08:21:25Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 d9ba56c22f22ae48ecc53629c2d49f1ae02fcbcb,"Flexible indexes refactoring notes (#4979) * add flexible indexes refactoring notes * include some of the PR comments in the notes * update / integrate items from the PR discussion * fill index creation, access, propagation sections * fill index use / encode / repr sections * fill remaining sections + tweaks * Update FLEXIBLE_INDEXES_NOTES.md Co-authored-by: Ryan Abernathey * Update FLEXIBLE_INDEXES_NOTES.md Co-authored-by: Ryan Abernathey * Update FLEXIBLE_INDEXES_NOTES.md Co-authored-by: Ryan Abernathey * Update FLEXIBLE_INDEXES_NOTES.md Co-authored-by: Ryan Abernathey * address review comments * address more review comments * [skip-ci] move notes to design_notes subdirectory Co-authored-by: Ryan Abernathey ",2021-03-17T16:47:29Z,2021-03-17T16:47:29Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 cb90d5542bd6868d5548ae8efb5815c249c2c329,"Fix html repr in untrusted notebooks (plain text fallback) (#4053) * add html pre element with text repr as fallback The PRE element is not displayed when CSS is injected. When CSS is not injected (untrusted notebook), the PRE element is shown but not the DIV container used for the HTML repr. * remove title elements in svg icons Prevent showing those when fallback to plain text repr. A title tag is already present in the HTML label elements. * add basic test * update what's new",2020-05-20T17:06:39Z,2020-05-20T17:06:39Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 63cc85759ac25605c8398d904d055df5dc538b94,add icomoon license (#3448),2019-10-25T15:40:46Z,2019-10-25T15:40:45Z,7c39fdcf60cc53af273aaf57e5babfd7c2fcd3db,0c7e9e762dbfd6554e60c953bf27493047d95109,13221727,4160723,2448579 1d0a2bc4970d9e7337fe307f4519bd936f7d7d89,"Detailed report for testing.assert_equal and testing.assert_identical (#1507) * more detailed AssertionError message for assert_identical * print differing dimensions/data/variables/attributes * minor tweaks * add what's new entry * add tests for diff_array_repr and diff_dataset_repr * pep8 * add differing dimensions in diff_array_repr * fix tests (explicit numpy dtypes) * fix tests (dtype shown / not shown in array repr) * minor tweaks",2019-01-18T09:16:30Z,2019-01-18T09:16:30Z,5d060630fae3d5a365a239dae12fe85ceed98a8d,cd792325681cbad9f663f2879d8b69f1edbb678f,13221727,4160723,19864447 846e28f8862b150352512f8e3d05bcb9db57a1a3,"DOC: move xarray related projects to top-level TOC section (#2357) * move xarray third-party projects to its own top-level TOC section * add hvplot",2018-08-10T20:13:08Z,2018-08-10T20:13:08Z,5d060630fae3d5a365a239dae12fe85ceed98a8d,caabe6633d9bef7f2f52807cc5a1f477f3a077d4,13221727,4160723,6815844 4ee244078ea90084624c1b6d006f50285f8f2d21,"DOC: add main sections to toc (#1946) * add main sections to toc * move whats new to ""help and references"" section",2018-02-27T19:04:24Z,2018-02-27T19:04:23Z,5d060630fae3d5a365a239dae12fe85ceed98a8d,f10b21bed2846b879806f87039b77245b18e7671,13221727,4160723,1217238 1a012080e0910f3295d0fc26806ae18885f56751,"Fix unexpected behavior of .set_index() since pandas 0.21.0 (#1723) * fix set_index behavior using pandas 0.21.0 * review comments",2017-11-17T00:54:50Z,2017-11-17T00:54:50Z,5d060630fae3d5a365a239dae12fe85ceed98a8d,5f199557d0f8f69fbea5e027a407146e2669a812,13221727,4160723, b8771934a2ef24fd3ce5a93fc2accb3f6fa12e4e,"xarray.core.variable.as_variable part of the public API (#1422) * as_variable now part of the public API * fully remove copy kwarg of as_variable * as_variable docstring changes * move as_variable to advanced API section * fix typo * add more tests for as_variable Also allow passing to `as_variable` objects that have a `data` attribute but no `values` attribute. * fix py27 test",2017-06-02T17:55:11Z,2017-06-02T17:55:11Z,5d060630fae3d5a365a239dae12fe85ceed98a8d,f10b21bed2846b879806f87039b77245b18e7671,13221727,4160723,1217238