issues

Problem description

Opening a file with open_mfdataset fails if dask is not installed: ```

xr.open_mfdataset('tests/sample-data/era5-levels-members.nc')

ModuleNotFoundError Traceback (most recent call last) ~/devel/MPY/xarray/xarray/backends/api.py in open_mfdataset(paths, chunks, concat_dim, compat, preprocess, engine, lock, data_vars, coords, autoclose, parallel, kwargs) 626 --> 627 datasets = [open_(p, open_kwargs) for p in paths] 628 file_objs = [getattr_(ds, '_file_obj') for ds in datasets] 629 if preprocess is not None:

~/devel/MPY/xarray/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs) 340 341 with close_on_error(store): --> 342 return maybe_decode_store(store) 343 else: 344 if engine is not None and engine != 'scipy':

~/devel/MPY/xarray/xarray/backends/api.py in maybe_decode_store(store, lock) 276 277 if chunks is not None: --> 278 from dask.base import tokenize 279 # if passed an actual file path, augment the token with 280 # the file modification time

ModuleNotFoundError: No module named 'dask.base' ```

Reason

The reason is that the test in maybe_decode_store is for None but open_mfdataset in line 614 sets chunks to {} if it evaluates to False.

In fact setting line https://github.com/pydata/xarray/blob/master/xarray/backends/api.py#L614 as follows solves the problem: open_kwargs = dict(engine=engine, chunks=chunks, lock=lock, autoclose=autoclose, **kwargs) I could not run the tests to see what breaks with this change as running pytest without dask fails with: ../cpython3/lib/python3.7/site-packages/py/_path/local.py:668: in pyimport __import__(modname) xarray/tests/__init__.py:105: in <module> if LooseVersion(dask.__version__) < '0.18': E AttributeError: module 'dask' has no attribute '__version__'

What happened?

The plugin interface warns with DeprecationWarning in python 3.10.

What did you expect to happen?

No warning to be issued.

Minimal Complete Verifiable Example

Python Running `pytest` on any *Xarray* importing test.

Relevant log output

Python .../python3.10/site-packages/xarray/backends/plugins.py:98: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select. entrypoints = entry_points().get("xarray.backends", ())

Anything else we need to know?

According to the Compatibity note in:

https://docs.python.org/3/library/importlib.metadata.html#entry-points

the new API is entry_points(group='xarray.backends')

Environment

• Related to #4309 and in particular: https://github.com/pydata/xarray/issues/4309#issuecomment-747055666
• [x] Passes isort . && black . && mypy . && flake8
• No user visible change

Once ZarrStore.open_group and NetCDF4DataStore.open accepts pathlib.Path objects we can remove _normalize_path from apiv2.open_dataset. All other backends accept them.

Note that zarr itself doesn't support them, yet. See: https://github.com/zarr-developers/zarr-python/issues/601

• [x] Related to #5232
• [x] ~Tests added~
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

@TomNicholas @dcherian & @pydata/xarray please review and propose changes

I'm at the item 6. of the how to release guide.

• [x] Closes #5073 (alternative to and more precise than #5237)
• [ ] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

@TomNicholas and all, sorry again for addressing this late.

Depending on the kind of problem python-eccdodes may return RuntimeError or ImportError.

• [x] Related to #5138
• [x] Passes pre-commit run --all-files

Is your feature request related to a problem? Please describe.

I started implementing a few backend plugins following the documentation so I noticed a few easily fixable quirks:

http://xarray.pydata.org/en/latest/internals/how-to-add-new-backend.html

The main one is the BackendEntrypoint is found deep into xarray.backends.common.

Describe the solution you'd like

I suggest to add an easier to find reference: * xarray.backends.BackendEntrypoint would do * but maybe even xarray.BackendEntrypoint.

Additional context

For the simplest plugins BackendEntrypoint would be the only import to connect to xarray.

See this dummy GDAL plugin: https://github.com/bopen/xarray-gdal/blob/main/xarray_gdal/xarray_plugin.py But also the rioxarray plugin in this PR: corteva/rioxarray/pull/281

• [x] Proposed fix for #5125
• [x] No need to add tests
• [x] Passes pre-commit run --all-files
• [x] Documentation is synced

This is mostly a stylistic revamp of the documentation on "How to add a new backend" after I followed the process a couple of times: * add import statments * use always my_*/My (there where some your_) * avoid ellipsis

The main code change is adding references to BackendEntrypoint and BackendArray in xarray.backends.

• [x] No issue, trivial cleanup
• [x] Tests modified to match

The behaviour is not too problematic as CF checkers don't complain about the empty string, but it is aesthetically annoying.

This happens more frequently after #2844 added the auxiliary variable to .coords.

• [x] Closes #5111
• [x] No need to add tests
• [x] Passes pre-commit run --all-files

What happened:

With current xarray master the rioxarray.open_rasterio crashes.

What you expected to happen:

With the released version 0.17.0 of xarray the open works as expected.

Minimal Complete Verifiable Example:

```python

import rioxarray rioxarray.open_rasterio("myfile.tif")

AttributeError Traceback (most recent call last) <ipython-input-7-fedc16472305> in <module> ----> 1 da = rioxarray.open_rasterio("../../../FLAIR-PROC/rome-lockdown-2020/georeferenced/20200309+0130+417_VV-20200109-20200115-EPSG:32632-5.human-activity.tif") 2 da

/usr/local/Caskroom/miniconda/base/envs/BOP/lib/python3.8/site-packages/rioxarray/_io.py in open_rasterio(filename, parse_coordinates, chunks, cache, lock, masked, mask_and_scale, variable, group, default_name, **open_kwargs) 820 821 da_name = attrs.pop("NETCDF_VARNAME", default_name) --> 822 data = indexing.LazilyOuterIndexedArray( 823 RasterioArrayWrapper( 824 manager,

AttributeError: module 'xarray.core.indexing' has no attribute 'LazilyOuterIndexedArray' ```

Anything else we need to know?:

LazilyOuterIndexedArray was recently renamed to LazilyIndexedArray by @aurghs in https://github.com/pydata/xarray/pull/4810 after a suggestion from @shoyer https://github.com/pydata/xarray/pull/4810#discussion_r586754165

Adding an alias would make the transition easier.

• [x] Closes #4309
• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

The documentation is in PR #4803

This is a big change so it better gets in early in the development cycle.

Some users visible changes to xr.open_dataset need to be discussed.

• [x] Related to #4309
• [x] Passes pre-commit run --all-files

Internal fix. At the moment even when setting XARRAY_BACKEND_API=v2 the defaults of open_dataset are defined by the v1 function. In APIv2 the defaults are defined by the backends (for example pseudonetcdf_.py sets mask_and_scale=False).

This PR fixes the backend defaults to be the same as in v1 in all cases.

• [x] Related to #4309
• [x] No tests added, all tests passing
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] No new functions/methods are listed in api.rst

You can now pass a close argument to xr.Dataset that closes the backend resources. This is only useful to backend developers, we need to add the documentation as per #4803 but it may need to be hidden and not documented to end-users.

The PR is draft pending some cleanups and the documentation.

• [x] Related to #4724
• [x] Tests sync'ed
• [x] Passes isort . && black . && mypy . && flake8
• [x] No user visible changes
• [x] No new functions/methods

This is a minor cleanup of the backend API v2.

• [x] Closes #4733
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• No new functions/methods are listed in api.rst

• [x] No issue, internal clenaup
• [x] Tests fixed
• [x] Passes isort . && black . && mypy . && flake8

Tiny cleanup to silence a couple CachingFileManager warnings about files not being properly closed.

• [x] Related to #4309
• [x] No tests added, all tests pass
• [x] Passes isort . && black . && mypy . && flake8
• [x] No user visible change
• [x] No new functions/methods

Stop importing api.py in apiv2.py and other internal cleanups.

This is needed by another change that results in a cyclic import due to api.

• [x] Relates to #4309
• [x] Tests updated
• [x] Passes isort . && black . && mypy . && flake8
• No user visible changes (including notable bug fixes) are documented in whats-new.rst
• No new functions/methods are listed in api.rst

Changes affects only APIv2.

This PR removes all remaining knowledge about what backend support from the generic code path in apiv2.open_dataset.

Once we make the switch to API v2 a big chunk of code can be removed from api.py.

• [x] Related to #4309
• No test added, but all tests pass.
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• No new functions/methods.

The change affects only APIv2.

Pros: - open_dataset passes the dataset description object filename_or_obj to the backends without fiddling with it. This is more and more important as several backends know how to open dicts, file's strs that are not paths (rasterio for example accepts GDAL strings that are not paths and that is one of the reasons engine="rasterio" cannot be used in open_dataset)

Cons: - at the moment backends expect paths to always be absolute, not sure if they use this info.

This is a small, pure code-refactor PR that removes some of the cruft left in open_dataset from #4187 and #4431

• [x] Additional clean ups after #4431
• [x] Pure code refactor, no change in tests
• [x] Passes isort . && black . && mypy . && flake8

Note that mode="r" is the default in open_group.

Always use the engine explicitly selected by the user and perform auto detection only when engine=None.

• [x] Closes #4457
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] No new functions/methods

The refactor complexity stems mostly from moving or duplicating some logic from _get_engine_from_magic_number to the backend stores. The current, rather ugly, code minimizes changes to the error conditions and messages.

Later in the backend refactor the plugins will provide the functionalities to identify the format from the file content and the duplication should be eliminated.

@jhamman do you think I need to acknowledge the CZI sponsorship in the changelog?

This ports the clean up of the messy if-else-chain from #3166. It should make adding the new back-end API easier.

The change is relatively trivial, but it touches code in xr.open_dataset so extra care should be used by reviewers. We think the change is safe and it does pass all tests.

• [x] Port the generic part of #3166
• [x] Tests sync'ed
• [x] Passes isort . && black . && mypy . && flake8
• [x] No user visible change
• [x] No new functions/methods

I cannot add reviewers myself, but we agreed this should go to @jhamman.

What happened:

When I open() a netCDF4 file and pass the file object to xr.open_dataset with engine='scipy' xarray silently ignores the value of the engine keyword and opens the file with h5netcdf. When opening an netCDF3 file with engine='h5netcdf' the scipy engine is used instead without warning.

What you expected to happen:

Silent autodetection of the engine is fine when passing engine=None, but it is very surprising when passing a value explicitly. I would have expected an exception or at least a warning.

Minimal Complete Verifiable Example:

```python

xr.open_dataset('netCDF4.nc', engine='scipy')

KeyError Traceback (most recent call last) [...] TypeError: Error: netCDF4.nc is not a valid NetCDF 3 file If this is a NetCDF4 file, you may need to install the netcdf4 library, e.g.,

        $ pip install netcdf4

fp = open('netCDF4.nc') xr.open_dataset(fp, engine='scipy') <xarray.Dataset> Dimensions: (line: 41587, pixel: 17751) Coordinates: * line (line) int64 0 1 2 3 4 5 ... 41582 41583 41584 41585 41586 * pixel (pixel) int64 0 1 2 3 4 5 ... 17746 17747 17748 17749 17750 Data variables: phase (line, pixel) float32 ... Attributes: Conventions: CF-1.7 [...] ```

• [x] Related but really orthogonal to #4309
• [x] Remove code, but keeps all existing tests passing (on MacOS with pandas=1.0.5)
• [x] Passes isort . && black . && mypy . && flake8
• [x] No user / developer visible changes are expected

@shoyer this could be the first pass of the simplification of the current backend API. It not needed for #4309, but it make more obvious that we'd be free to keep the AbstractDataStore class if we wanted to.

The reason the change can be considered reasonably innocuous is that warnings would have been printed on access to any of the removed methods, but you never know. Your call (but I'd love to have a delete-only contribution to xarray :D).

Please note that: 1. this change is the absolute lowest hanging fruit, 2. removing __enter__ and __exit__ appears feasible as they are used only in tests, but it is a slightly bigger change, 3. removing load is marginally more complex because it handle the special case when a variable name is equal to None.

This is a minor cleanup of the conda environment creation files: * no direct reference to ecCodes * conda-only dependency resolution

This is a minor cleanup of the backend_kwargs hack that was needed to support CachingFileManager and that I forgot to remove after I stopped using it.

I didn't include this bit in #2540 because I didn't want to pollute the important fix with cleanup noise.

This is a trivial fix to remove the use of the mode keyword in the cfgrib.open_file call. The keyword has always be ignored and will be deprecated with the next version of cfgrib.

Better to have this fix in before the v0.11 release.

This is currently a WIP PR for review. - [x] Addresses #2475 - only read support is proposed for now. - [x] Tests added. - [x] Documented, including whats-new.rst for all changes and api.rst for new API.

The implementation depends on the python module cfgrib and the C-library ecCodes to be installed.

Work in progress items: - [x] the coordinate rename doesn't really belong here, move it to cfgrib, - [x] port cfgrib backend to use the new CachingFileManager interface - [x] implement proper locking - [x] test dask support (real performance with dask depends on saving the external index) - ~~~the encoding options are needlessly different from the standard ones, try to sync them~~~ not urgent

cc @StephanSiemen @iainrussell

Fix TestCfGrib broken by a user-visible backward-incompatible change in cfgrib>=0.9.2.

No other user visible coordinate name change is expected so the test can remain like it is.

Sorry for the inconvenience!

• [x] Noted in the discussion of #2500
• [x] Tests added
• ~~Fully documented, including whats-new.rst for all changes and api.rst for new API~~

_get_default_engine chooses the engine suitable to open the path, ATM it only detects URL's. In this PR I propose to also detect GRIB files by the extension and suggest to the user which engine to use.

Updated: The whole engine detection strategy looked a bit fragile as the .gz extension was recognised outside of _get_default_engine and the nested ifs and try / except blocks make the code poorly readable. I added a bit of a refactor in this area.

• ~~Closes xxxx~~
• [x] Tests added
• ~~Fully documented, including whats-new.rst for all changes and api.rst for new API.~~ No change to document.

The cfgrib project offers to open a GRIB file as an xarray.Dataset: ```

import cfgrib cfgrib.open_dataset('era5-levels-members.grib') <xarray.Dataset> Dimensions: (air_pressure: 2, latitude: 61, longitude: 120, number: 10, time: 4) Coordinates: * number (number) int64 0 1 2 3 4 5 6 7 8 9 * time (time) datetime64[ns] 2017-01-01 ... 2017-01-02T12:00:00 step timedelta64[ns] ... * air_pressure (air_pressure) float64 850.0 500.0 * latitude (latitude) float64 90.0 87.0 84.0 81.0 ... -84.0 -87.0 -90.0 * longitude (longitude) float64 0.0 3.0 6.0 9.0 ... 351.0 354.0 357.0 valid_time (time) datetime64[ns] ... Data variables: z (number, time, air_pressure, latitude, longitude) float32 ... t (number, time, air_pressure, latitude, longitude) float32 ... Attributes: GRIB_edition: 1 GRIB_centre: ecmf GRIB_centreDescription: European Centre for Medium-Range Weather Forecasts GRIB_subCentre: 0 history: GRIB to CDM+CF via cfgrib-0.9.../ecCodes-2... It would be very helpful to have a backend for it distributed with *xarray* to be selected via the `engine` keyword argument: import xarray as xr xr.open_dataset('era5-levels-members.grib', engine='cfgrib') <xarray.Dataset> Dimensions: (air_pressure: 2, latitude: 61, longitude: 120, number: 10, time: 4) Coordinates: * number (number) int64 0 1 2 3 4 5 6 7 8 9 * time (time) datetime64[ns] 2017-01-01 ... 2017-01-02T12:00:00 step timedelta64[ns] ... * air_pressure (air_pressure) float64 850.0 500.0 * latitude (latitude) float64 90.0 87.0 84.0 81.0 ... -84.0 -87.0 -90.0 * longitude (longitude) float64 0.0 3.0 6.0 9.0 ... 351.0 354.0 357.0 valid_time (time) datetime64[ns] ... Data variables: z (number, time, air_pressure, latitude, longitude) float32 ... t (number, time, air_pressure, latitude, longitude) float32 ... Attributes: GRIB_edition: 1 GRIB_centre: ecmf GRIB_centreDescription: European Centre for Medium-Range Weather Forecasts GRIB_subCentre: 0 history: GRIB to CDM+CF via cfgrib-0.9.../ecCodes-2... ```

cc @StephanSiemen @iainrussell