issue_comments

I think it is OK to add a dedicated xarray.EngineGuessingError, but I'd not try anything more complex and in general invite users to pass engine explicitly.

I'm reluctantly inclined to agree. (But "reluctantly" only because this means that we may have a lot of xcube code to check and update now.) If the engine is explicitly specified, the question of which exceptions to expect becomes a lot easier to handle, since it's now just between the caller and the engine.

Probably the open_dataset documentation should actively discourage reliance on engine guessing for anything but interactive use, and point out that callers need to check the documentation for the selected engine to determine which exceptions (other than EngineGuessingError) might be thrown.

Thanks for raising an issue @pont-us. I believe this may have been fixed by #5271 (which was merged just after the 0.18.0 release). Could you install the development version of xarray and try again to see if the problem persists?

Thanks, I've just tested the current head (24c61522) with both the minimal example above and the xcube unit test that was failing with dask 2021.05.0, and can confirm that the error no longer occurs.

@aurghs Point taken, but in that case the xarray.open_dataset documentation probably needs updating: currently it says ‘Strings and Path objects are interpreted as a path to a netCDF file or an OpenDAP URL and opened with python-netCDF4, unless the filename ends with .gz…’. In particular I find the behaviour with Path confusing: this type is unambiguously a filesystem path, but still results in a ValueError rather than FileNotFoundError. And in Python 3, str and bytes are distinct (though I suppose a str might still be a URL rather than a file).

In principle, though, I'd be OK with any selection of potential exceptions, as long as they're documented. The current practical problem for us (the xcube developers) is that the behaviour change from 0.17 to 0.18 means that we now have to audit our existing codebases for potential uncaught ValueErrors.

Could you kindly share the Python traceback from the error message?

Sorry, that was remiss of me. Here's the output from the test script above, including the traceback.

$ python test_append_bytes.py Creating |U1 Zarr... Appending to |U1 Zarr with Zarr library... Appending to |U1 Zarr with xarray... Creating |S1 Zarr... Appending to |S1 Zarr with Zarr library... Appending to |S1 Zarr with xarray... Traceback (most recent call last): File "/home/pont/test_append_bytes.py", line 21, in <module> test_append("|S1", "test-s.zarr") File "/home/pont/test_append_bytes.py", line 18, in test_append ds_to_append.to_zarr(zarr_path, mode="a") File "/home/pont/loc/repos/xarray/xarray/core/dataset.py", line 1877, in to_zarr return to_zarr( File "/home/pont/loc/repos/xarray/xarray/backends/api.py", line 1414, in to_zarr _validate_datatypes_for_zarr_append(dataset) File "/home/pont/loc/repos/xarray/xarray/backends/api.py", line 1261, in _validate_datatypes_for_zarr_append check_dtype(k) File "/home/pont/loc/repos/xarray/xarray/backends/api.py", line 1252, in check_dtype raise ValueError( ValueError: Invalid dtype for data variable: <xarray.DataArray 'x' ()> array(b'', dtype='|S1') dtype must be a subtype of number, datetime, bool, a fixed sized string, a fixed size unicode string or an object

Thank you very much for the swift fix -- I had been meaning to put a PR together but things have been very busy lately.