When open_dataset() fails because no working engines are found, it suggests installing the dependencies of the compatible internal backends, providing explicitly the list. - [x] closes #5302 - [x] Tests added - [x] Passes pre-commit run --all-files

Fix open_dataset regression, expands ~ in filepath_or_obj when necessary. I have checked the behaviour of the engines. It seems that pynio already expands ~.

• [x] Closes #5098
• [x] Passes pre-commit run --all-files

Currently, a broken engine breaks xarray.open_dataset. I have added a try except to avoid this problem.

Old behaviour:

```python

ds = xr.open_dataset('example.nc') Traceback (most recent call last):

File "/usr/local/Caskroom/miniconda/base/envs/xarray/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code exec(code_obj, self.user_global_ns, self.user_ns)

File "<ipython-input-3-0c694cae8262>", line 1, in <module> arr = xr.open_dataset("example.nc")

File "/Users/barghini/devel/xarray/xarray/backends/api.py", line 495, in open_dataset backend = plugins.get_backend(engine)

File "/Users/barghini/devel/xarray/xarray/backends/plugins.py", line 115, in get_backend engines = list_engines()

File "/Users/barghini/devel/xarray/xarray/backends/plugins.py", line 97, in list_engines return build_engines(pkg_entrypoints)

File "/Users/barghini/devel/xarray/xarray/backends/plugins.py", line 84, in build_engines external_backend_entrypoints = backends_dict_from_pkg(pkg_entrypoints)

File "/Users/barghini/devel/xarray/xarray/backends/plugins.py", line 58, in backends_dict_from_pkg backend = pkg_ep.load()

File "/usr/local/Caskroom/miniconda/base/envs/xarray/lib/python3.8/site-packages/pkg_resources/init.py", line 2450, in load return self.resolve()

File "/usr/local/Caskroom/miniconda/base/envs/xarray/lib/python3.8/site-packages/pkg_resources/init.py", line 2456, in resolve module = import(self.module_name, fromlist=['name'], level=0)

File "/Users/barghini/devel/xarray-sentinel/xarray_sentinel/sentinel1.py", line 13 ERROR ^ SyntaxError: invalid syntax ```

New behaviour:

```python

ds = xr.open_dataset('example.nc') /Users/barghini/devel/xarray/xarray/backends/plugins.py:61: RuntimeWarning: Engine sentinel-1 loading failed: name 'ERROR' is not defined warnings.warn(f"Engine {name} loading failed:\n{ex}", RuntimeWarning) ```

• [x] Tests added
• [x] Passes pre-commit run --all-files

• add backend documentation

• rename store_spec in filename_or_objin backend entrypoint method guess_can_open

• [x] Related #4803

Ensure that backend list are always sorted in the same way. In particular: - the standards backend are always the first in the following order: "netcdf4", "h5netcdf", "scipy" - all the other backends a sorted in lexicographic order.

the changes involve two files (plugins.py and test_plugins.py) and they include: - add utility function for sorting backends sort_backends - Update tests - Small changes in variables/functions names.

• [x] Tests added
• [x] Passes pre-commit run --all-files

Port all the engines to the new API apiv2. Note: - test_autoclose_future_warning has been removed because in apiv2.py autoclose has been removed - in open_backend_dataset_psedonetcdf currently is still used **format_kwargs and the signature is defined explicitly

• [x] Related to https://github.com/pydata/xarray/issues/4309
• [x] Tests updated
• [x] Passes isort . && black . && mypy . && flake8

Remove close_on_error in store.py. This change involves only apiv2. Currently, api_v2.open_dataset can take in input a store instead of a file. In case of error, xarray closes the store. Xarray should manage the closure of a store that has been instantiated externally. This PR correct this behaviour in apiv2

• [x] Related https://github.com/pydata/xarray/pull/4673
• [x] Passes isort . && black . && mypy . && flake8

This PR aims to avoid conflicts during the transition period between the old backend implementation and the new plugins. During the transition period will coexist both external backend plugins and internal ones. Currently, if two plugins with the same name are detected, we just pick one randomly. It would be better to be sure to use the external one.

Main changes: - Remove from setup.cfg - Store in the internal backend and stored in the dictionary in plugins.py. The dictionary is updated with the external plugins detected by pkg_resources. - Move the class BackendEntrypoints in common.py to resolve a circular import.
- Add a test

• [x] Related to https://github.com/pydata/xarray/issues/4309
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8

This PR fixes https://github.com/pydata/xarray/issues/4708. It's a very small change. Changes: - In dataset._check_chunks_compatibility now it doesn't raise a warning if the last chunk % preferred_chunk != 0. - Update tests - Style: rename a variable inside dataset._check_chunks_compatibility

• [x] Closes https://github.com/pydata/xarray/issues/4708
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8

• 4646 add tests on chunking without using a with statement, causing unexpected warnings.

• Add filterwarnings in test_plugins.test_remove_duplicates tests and backend_tests.test_chunking_consintency
• [x] Tests fixex
• [x] Passes isort . && black . && mypy . && flake8

This PR remove autoclose option from open_dataset (both api.py and apiv2.py) and the corresponding test test_autoclose_future_warning from test.py autoclose=True option was deprecated in https://github.com/pydata/xarray/pull/2261 since xarray now uses a LRU cache to manage open file handles.

• [x] Related to https://github.com/pydata/xarray/issues/4309 and https://github.com/pydata/xarray/pull/2261,
• [x] Tests updated
• [x] Passes isort . && black . && mypy . && flake8

Related to https://github.com/pydata/xarray/issues/4496 Add encodings["preferred_chunks"] in zarr, used in open_dataset instead of encodings["chunks"].

• [x] Related to #https://github.com/pydata/xarray/issues/4496
• [x] Passes isort . && black . && mypy . && flake8

It's the last part of, and closes #4595. Here we unify the code for chunking in apiv2.open_dataset. Note the code unification is only a refactor, there aren't functional changes since the zarr chunking has been already aligned with the others.

• [x] Related to https://github.com/pydata/xarray/issues/4496
• [x] Passes isort . && black . && mypy . && flake8

Part of https://github.com/pydata/xarray/pull/4595 The changes involve only open_dataset(..., engine=zarr) (and marginally open_zarr), in particular, _get_chunks has been modified to fit #4496 (comment) option 1 chunking behaviour and align open_dataset chunking with dataset.chunk:

• with auto it uses dask auto-chunking (if a preferred_chunking is defined, dask will take it into account as done in dataset.chunk)
• with -1 it uses dask but no chunking.
• with {} it uses the backend encoded chunks (when available) for on-disk data (xr.open_dataset) and the current chunking for already opened datasets (ds.chunk)

Add some test - [x] Releted to pydata#4496 - [x] Tests added - [x] Passes isort . && black . && mypy . && flake8 - [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

The aim is to split the PR https://github.com/pydata/xarray/pull/4595 in small PRs. In this smaller PR there aren't changes in xarry interfaces, it's only a small code refactor: - Move get_chunks from zarr.py to dataset.py - Align apiv2 to apiv1: in apiv2 replace zarr.ZarrStore.maybe_chunk with dataset._maybe_chunk and zarr.ZarrStore.get_chunk with dataset._get_chunks. - remove zarr.ZarrStore.maybe_chunk and zarr.ZarrStore.get_chunks (no more used)

• [x] Related #4496
• [x] Passes isort . && black . && mypy . && flake8
• No user visible changes (including notable bug fixes) are documented in whats-new.rst
• No new functions/methods are listed in api.rst

Proposal for the new API of open_dataset(). It is implemented in apiv2.py and it doesn't modify the current behavior of api.open_dataset(). It is something in between the first and second alternative suggested at https://github.com/pydata/xarray/issues/4490#issue-715374721, see the related quoted text:

Describe alternatives you've considered

For the overall approach:

1. We could keep the current design, with separate keyword arguments for decoding options, and just be very careful about passing around these arguments. This seems pretty painful for the backend refactor, though.
2. We could keep the current design only for the user facing open_dataset() interface, and then internally convert into the DecodingOptions() struct for passing to backend constructors. This would provide much needed flexibility for backend authors, but most users wouldn't benefit from the new interface. Perhaps this would make sense as an intermediate step?

Instead of a class for the decoders, I have added a function: resolve_decoders_kwargs.
resolve_decoders_kwargs performs two tasks: - If decode_cf is False, it sets to False all the decoders supported by the backend (using inspect). - It filters out the None decoder keywords.

So xarray manages the keyword decode_cf and passes on only the non-default decoders to the backend. If the user sets to a non-None value a decoder not supported by the backend, the backend will raise an error.

With this implementation drop_variable should be always supported by the backend. But I think this could be implemented easely by all the backends. I wouldn't group it with the decoders: to me, it seems to be more a filter rather than a decoder.

The behavior decode_cf is unchanged.

PRO: - the user doesn't need to import and instantiate a class. - users get the argument completion on open_dataset. - the backend defines directly in open_backend_dataset_${engine} API the accepted decoders. - xarray manages decode_cf, not the backends.

Missing points: - deccode_cf should be renamed decode. Probably, the behavior of decode should be modified for two reason: - currently If decode_cf is False, it sets the decoders to False, but there is no check on the other values. The accepted values should be: None (it keeps decoders default values), True (it sets all the decoders to True), False (it sets all the decoders to False). - currently we can set both a decoder and decode_cf without any warning. , but the - Deprecate backend_kwargs (or kwargs). - Separate mask_and_scale?

I think that we need a different PR for the three of them.

• [x] related to https://github.com/pydata/xarray/issues/4490#
• [x] Passes isort . && black . && mypy . && flake8

Currently, the interface between the backend and xarray is the class/container BackendEntrypoint, that must be instantiated by the backend. With this pull request, BackendEntrypoint is replaced by AbstractBackendEntrypoint. The backend will inherit from this class.

Reason for these changes: - This type of interface is more standard.

• [x] Tests updated
• [x] Passes pre-commit run --all-files

Currently list_engines returns the list of all installed backend plus the list of the internal ones. For the internal ones, there is no check on the installed dependencies. Now the registration of the internal backends is done by the backends only if the needed dependencies are installed.

• [x] Passes pre-commit run --all-files

• It's an update of @jhamman pull request https://github.com/pydata/xarray/pull/3166
• It uses entrypoints module to detect the installed engines. The detection is done at open_dataset function call and it is cached. It raises a warning in case of conflicts.
• Add a class for the backend interface BackendEtrypoint instead of a function.

Modified files: - add plugins.py containing detect_engines function and BackendEtrypoint. - dependencies file to add entrypoints. - backend.init to add detect_engines - apiv2.py and api.py do use detect_engines - zarr.py, h5netcdf_.py, cfgrib.py to instatiate the BackendEtrypoint.

• [x] Related to #3166
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8

The aim is to split the PR #4595 in small PRs. The scope of this smaller PR is to modify the default of chunks in dataset.chunk to align the behaviour to xr.open_dataset. The main changes are: - Modify the default of chunk in dataset.chunk, datarray.chunk and variable.chunk from None to {}. - If the user pass chunk=None inside is set to {} - Add a future warning to advice that the usage of None will raise an error in the future.

Note that the changes currently don't modify the behaviour of dataset.chunk

• [x] Related #4496
• [x] Passes isort . && black . && mypy . && flake8

Related to PR https://github.com/pydata/xarray/pull/4595. In this smaller PR, there aren't changes functional changes, it's only a small code refactor needed to simplify pydata#4595. Changes in apiv2.dataset_from_backend_dataset: - rename ds in backend_ds and ds2 in ds. - simplify if in chunking and split code adding function _chunks_ds - add _get_mtime specific function

Make resolve_decoders_kwargs and dataset_from_backend_dataset private - [x] related to https://github.com/pydata/xarray/pull/4595 - [x] Passes isort . && black . && mypy . && flake8

This work aims to harmonize the way zarr deals with chunking to have similar behavior for all other backends and unify the code. Most of the changes involve the new API, apiv2.py, except for some changes in the code that has been added with the merge of https://github.com/pydata/xarray/pull/4187.

main changes: - refactor apiv2.dataset_from_backend_dataset function. - move _get_chunks from zarr to dataset. - modify _get_chunks to fit https://github.com/pydata/xarray/issues/4496#issuecomment-720785384 option 1 chunking behaviuor - Add warning when it is used in ds.chunk(..., chunk=None) - Add some test

nedded separate pull request for the following missing points: - standardize the key in encodings to define the on-disk chunks: chunksizes - add a specific key in encodings for preferred chunking (currently it is used chunks)

• [x] Related https://github.com/pydata/xarray/issues/4496
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

This work aims to harmonize the way zarr deals with chunking to have similar behavior for all other backends and unify the code. Most of the changes involve the new API, apiv2.py, except for some changes in the code that has been added with the merge of https://github.com/pydata/xarray/pull/4187.

main changes: - refactor apiv2.dataset_from_backend_dataset function. - move get_chunks from zarr to dataset.

current status: - in apiv2.open_dataset chunks='auto' and chunks={} now has the same beahviuor - in apiv2.open_dataset for all the backends now the default chunking is provided by the backend, if it is not available it uses one big chunk.

Missing points: - standardize the key in encodings to define the on-disk chunks: chunksizes - add a specific key in encodings for preferred chunking (currently it is used chunks)

There is one open point to be discussed yet: dataset.chunks and open_dataset(..., chunks=...) have different behaviors. dataset.chunks(chunks={}) opens the dataset with only one chunk per variable, while in open_dataset(..., chunks={}) it uses encodings['chunks'], when available.

Note that also chunks=None has a different behaviour: open_dataset(..., chunks=None) (or open_dataset(...), it's the deafult) returns variables without chunks, while dataset.chunk(chunks=None) (or dataset.chunk(), it's the default) has the same behavior of dataset.chunk(chunks=None). Probably it's not worth changing it.

• [x] related to https://github.com/pydata/xarray/issues/4496
• [ ] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

I propose this small change with a view to unifying in open_dataset the logic of zarr chunking with that of the other backends. Currently, the function maybe_chunk is duplicated: it is defined inside the function dataset.chunks and as method of zarr.ZarrStore. This last function has been added with the recent merge of #4187 . I merged the two functions in a private function _maybe_chunk inside the module dataset.

• [x] Addresses #4309
• [ ] Tests added
• [x] Passes isort . && black . && mypy . && flake8

The first draft of the new backend API: - Move decoding inside the bakends. - Backends return Dataset with BackendArray. - Xarray manages chunking and caching. - Some code is duplicated, it will be simplified later. - Zarr chunking is still inside the backend for now.

cc @jhamman @shoyer

• [x] Addresses #4309
• [ ] Tests added
• [ ] Passes isort . && black . && mypy . && flake8
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Releated to ticket #1621. Add decode_timedelta kwargs in open_dataset, xr.open_datarray, xr.open_zarr, xr.decode_cf. If not passed explicitly the behaviour is not changed. - [x] Tests added for xr.decode_cf. - [x] Passes isort -rc . && black . && mypy . && flake8 - [x] Fully documented, including whats-new.rst for all changes and api.rst for new API