issues

• Related to #4309 and in particular: https://github.com/pydata/xarray/issues/4309#issuecomment-747055666
• [x] Passes isort . && black . && mypy . && flake8
• No user visible change

Once ZarrStore.open_group and NetCDF4DataStore.open accepts pathlib.Path objects we can remove _normalize_path from apiv2.open_dataset. All other backends accept them.

Note that zarr itself doesn't support them, yet. See: https://github.com/zarr-developers/zarr-python/issues/601

• [x] Related to #5232
• [x] ~Tests added~
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

@TomNicholas @dcherian & @pydata/xarray please review and propose changes

I'm at the item 6. of the how to release guide.

• [x] Closes #5073 (alternative to and more precise than #5237)
• [ ] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

@TomNicholas and all, sorry again for addressing this late.

Depending on the kind of problem python-eccdodes may return RuntimeError or ImportError.

• [x] Related to #5138
• [x] Passes pre-commit run --all-files

• [x] Proposed fix for #5125
• [x] No need to add tests
• [x] Passes pre-commit run --all-files
• [x] Documentation is synced

This is mostly a stylistic revamp of the documentation on "How to add a new backend" after I followed the process a couple of times: * add import statments * use always my_*/My (there where some your_) * avoid ellipsis

The main code change is adding references to BackendEntrypoint and BackendArray in xarray.backends.

• [x] No issue, trivial cleanup
• [x] Tests modified to match

The behaviour is not too problematic as CF checkers don't complain about the empty string, but it is aesthetically annoying.

This happens more frequently after #2844 added the auxiliary variable to .coords.

• [x] Closes #5111
• [x] No need to add tests
• [x] Passes pre-commit run --all-files

• [x] Closes #4309
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

The documentation is in PR #4803

This is a big change so it better gets in early in the development cycle.

Some users visible changes to xr.open_dataset need to be discussed.

• [x] Related to #4309
• [x] Passes pre-commit run --all-files

Internal fix. At the moment even when setting XARRAY_BACKEND_API=v2 the defaults of open_dataset are defined by the v1 function. In APIv2 the defaults are defined by the backends (for example pseudonetcdf_.py sets mask_and_scale=False).

This PR fixes the backend defaults to be the same as in v1 in all cases.

• [x] Related to #4309
• [x] No tests added, all tests passing
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] No new functions/methods are listed in api.rst

You can now pass a close argument to xr.Dataset that closes the backend resources. This is only useful to backend developers, we need to add the documentation as per #4803 but it may need to be hidden and not documented to end-users.

The PR is draft pending some cleanups and the documentation.

• [x] Related to #4724
• [x] Tests sync'ed
• [x] Passes isort . && black . && mypy . && flake8
• [x] No user visible changes
• [x] No new functions/methods

This is a minor cleanup of the backend API v2.

• [x] Closes #4733
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• No new functions/methods are listed in api.rst

• [x] No issue, internal clenaup
• [x] Tests fixed
• [x] Passes isort . && black . && mypy . && flake8

Tiny cleanup to silence a couple CachingFileManager warnings about files not being properly closed.

• [x] Related to #4309
• [x] No tests added, all tests pass
• [x] Passes isort . && black . && mypy . && flake8
• [x] No user visible change
• [x] No new functions/methods

Stop importing api.py in apiv2.py and other internal cleanups.

This is needed by another change that results in a cyclic import due to api.

• [x] Relates to #4309
• [x] Tests updated
• [x] Passes isort . && black . && mypy . && flake8
• No user visible changes (including notable bug fixes) are documented in whats-new.rst
• No new functions/methods are listed in api.rst

Changes affects only APIv2.

This PR removes all remaining knowledge about what backend support from the generic code path in apiv2.open_dataset.

Once we make the switch to API v2 a big chunk of code can be removed from api.py.

• [x] Related to #4309
• No test added, but all tests pass.
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• No new functions/methods.

The change affects only APIv2.

Pros: - open_dataset passes the dataset description object filename_or_obj to the backends without fiddling with it. This is more and more important as several backends know how to open dicts, file's strs that are not paths (rasterio for example accepts GDAL strings that are not paths and that is one of the reasons engine="rasterio" cannot be used in open_dataset)

Cons: - at the moment backends expect paths to always be absolute, not sure if they use this info.

This is a small, pure code-refactor PR that removes some of the cruft left in open_dataset from #4187 and #4431

• [x] Additional clean ups after #4431
• [x] Pure code refactor, no change in tests
• [x] Passes isort . && black . && mypy . && flake8

Note that mode="r" is the default in open_group.

Always use the engine explicitly selected by the user and perform auto detection only when engine=None.

• [x] Closes #4457
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] No new functions/methods

The refactor complexity stems mostly from moving or duplicating some logic from _get_engine_from_magic_number to the backend stores. The current, rather ugly, code minimizes changes to the error conditions and messages.

Later in the backend refactor the plugins will provide the functionalities to identify the format from the file content and the duplication should be eliminated.

@jhamman do you think I need to acknowledge the CZI sponsorship in the changelog?

This ports the clean up of the messy if-else-chain from #3166. It should make adding the new back-end API easier.

The change is relatively trivial, but it touches code in xr.open_dataset so extra care should be used by reviewers. We think the change is safe and it does pass all tests.

• [x] Port the generic part of #3166
• [x] Tests sync'ed
• [x] Passes isort . && black . && mypy . && flake8
• [x] No user visible change
• [x] No new functions/methods

I cannot add reviewers myself, but we agreed this should go to @jhamman.

• [x] Related but really orthogonal to #4309
• [x] Remove code, but keeps all existing tests passing (on MacOS with pandas=1.0.5)
• [x] Passes isort . && black . && mypy . && flake8
• [x] No user / developer visible changes are expected

@shoyer this could be the first pass of the simplification of the current backend API. It not needed for #4309, but it make more obvious that we'd be free to keep the AbstractDataStore class if we wanted to.

The reason the change can be considered reasonably innocuous is that warnings would have been printed on access to any of the removed methods, but you never know. Your call (but I'd love to have a delete-only contribution to xarray :D).

Please note that: 1. this change is the absolute lowest hanging fruit, 2. removing __enter__ and __exit__ appears feasible as they are used only in tests, but it is a slightly bigger change, 3. removing load is marginally more complex because it handle the special case when a variable name is equal to None.

This is a minor cleanup of the conda environment creation files: * no direct reference to ecCodes * conda-only dependency resolution

This is a minor cleanup of the backend_kwargs hack that was needed to support CachingFileManager and that I forgot to remove after I stopped using it.

I didn't include this bit in #2540 because I didn't want to pollute the important fix with cleanup noise.

This is a trivial fix to remove the use of the mode keyword in the cfgrib.open_file call. The keyword has always be ignored and will be deprecated with the next version of cfgrib.

Better to have this fix in before the v0.11 release.

This is currently a WIP PR for review. - [x] Addresses #2475 - only read support is proposed for now. - [x] Tests added. - [x] Documented, including whats-new.rst for all changes and api.rst for new API.

The implementation depends on the python module cfgrib and the C-library ecCodes to be installed.

Work in progress items: - [x] the coordinate rename doesn't really belong here, move it to cfgrib, - [x] port cfgrib backend to use the new CachingFileManager interface - [x] implement proper locking - [x] test dask support (real performance with dask depends on saving the external index) - ~~~the encoding options are needlessly different from the standard ones, try to sync them~~~ not urgent

cc @StephanSiemen @iainrussell

Fix TestCfGrib broken by a user-visible backward-incompatible change in cfgrib>=0.9.2.

No other user visible coordinate name change is expected so the test can remain like it is.

Sorry for the inconvenience!

• [x] Noted in the discussion of #2500
• [x] Tests added
• ~~Fully documented, including whats-new.rst for all changes and api.rst for new API~~

_get_default_engine chooses the engine suitable to open the path, ATM it only detects URL's. In this PR I propose to also detect GRIB files by the extension and suggest to the user which engine to use.

Updated: The whole engine detection strategy looked a bit fragile as the .gz extension was recognised outside of _get_default_engine and the nested ifs and try / except blocks make the code poorly readable. I added a bit of a refactor in this area.

• ~~Closes xxxx~~
• [x] Tests added
• ~~Fully documented, including whats-new.rst for all changes and api.rst for new API.~~ No change to document.