As discussed in #8322, here is the test for implementing the vectorization

Only this test seems to fail in test_coding_times.py : https://github.com/pydata/xarray/blob/f895dc1a748b41d727c5e330e8d664a8b8780800/xarray/tests/test_coding_times.py#L1061-L1071

I don't really understand why though if you have an idea

Is your feature request related to a problem?

I developped a rust based project with python bindings code to parse the CF time conventions and deal with datetime operations.

You can find the project on GitHub at https://github.com/antscloud/cftime-rs.

It was something missing in the rust ecosystem to deal with NetCDF files, As the project in Rust hits its first working version, I wanted to explore the maturinecosystem and the Rust as a backend for python code. I ended up creating a new cftime implementation for python that have significant perfomance improvement (somewhere between 4 times to 20 times faster depending on the use case) over cftimeoriginal Cython code.

There are surely missing features compared to cftime and need to be tested more, but I think it could be interested as a replacement for some xarray operations (mainly for speed) regarding some of the issues of topic-cftime label

Please, let me know if xarray team could be interested. If you are, I can open a pull request to see it is possible, where it breaks the unit tests and if it's worth it

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

As discussed in #8302, here is a first attempt to implement cftime_rs.

There are a lot of tests and I struggle to understand all the processing in coding/times.py. However, with this first attempt I've been able to make the test_cf_datetime work (ignoring one test)

https://github.com/pydata/xarray/blob/8423f2c47306cc3a4a52990818964f278179491f/xarray/tests/test_coding_times.py#L127-L131

Also there are some key differences betwwen cftime and cftime-rs : - A long int is used to represent the timestamp internally, so cftime-rs will not overflow as soon as numpy, pythonor cftime. It can go from -291,672,107,014 BC to 291,672,107,014 AD approximately and this depends on calendar. - There is no only_use_python_datetimes argument. Instead there are 4 distinct functions : - date2num() - num2date() - num2pydate() - pydate2num() - These functions only take a python list of one dimension and return a list of one dimension. A conversion should be done before. - There is no multiple datetime type (there are hidden) but instead a single object PyCFDatetime - There is no conda repository at the moment

Finally, and regardless of this PR, I guess there could be a speed improvement by vectorizing operations by replacing this : https://github.com/pydata/xarray/blob/df0ddaf2e68a6b033b4e39990d7006dc346fcc8c/xarray/coding/times.py#L622-L649

by something like this :

https://github.com/pydata/xarray/blob/8423f2c47306cc3a4a52990818964f278179491f/xarray/coding/times.py#L631-L670

We can use numpy instead of list comprehensions. It takes a bit more of memory though.

What happened?

I found myself using a netcdf with empty units and by using xarray i was unable to use open_dataset due to the parsing of cf conventions. I reproduce the bug, and it happens in a particular situation when the units is an empty list (See Minimal Complete Verifiable Example)

What did you expect to happen?

To parse the units attribute as an empty string ?

Minimal Complete Verifiable Example

```Python temp = 15 + 8 * np.random.randn(2, 2, 3) precip = 10 * np.random.rand(2, 2, 3) lon = [[-99.83, -99.32], [-99.79, -99.23]] lat = [[42.25, 42.21], [42.63, 42.59]]

for real use cases, its good practice to supply array attributes such as

units, but we won't bother here for the sake of brevity

ds = xr.Dataset( { "temperature": (["x", "y", "time"], temp), "precipitation": (["x", "y", "time"], precip), }, coords={ "lon": (["x", "y"], lon), "lat": (["x", "y"], lat), "time": pd.date_range("2014-09-06", periods=3), "reference_time": pd.Timestamp("2014-09-05"), }, ) ds.temperature.attrs["units"] = []

ds.to_netcdf("test.nc")

ds = xr.open_dataset("test.nc") ds.close() ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python

TypeError Traceback (most recent call last) Input In [3], in <cell line: 1>() ----> 1 ds = xr.open_dataset("test.nc") 2 print(ds["temperature"].attrs) 3 ds.close()

File ~/.local/src/miniconda/envs/uptodatexarray/lib/python3.10/site-packages/xarray/backends/api.py:495, in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, backend_kwargs, args, kwargs) 483 decoders = _resolve_decoders_kwargs( 484 decode_cf, 485 open_backend_dataset_parameters=backend.open_dataset_parameters, (...) 491 decode_coords=decode_coords, 492 ) 494 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None) --> 495 backend_ds = backend.open_dataset( 496 filename_or_obj, 497 drop_variables=drop_variables, 498 decoders, 499 kwargs, 500 ) 501 ds = _dataset_from_backend_dataset( 502 backend_ds, 503 filename_or_obj, (...) 510 *kwargs, 511 ) 512 return ds

File ~/.local/src/miniconda/envs/uptodatexarray/lib/python3.10/site-packages/xarray/backends/netCDF4_.py:564, in NetCDF4BackendEntrypoint.open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, format, clobber, diskless, persist, lock, autoclose) 562 store_entrypoint = StoreBackendEntrypoint() 563 with close_on_error(store): --> 564 ds = store_entrypoint.open_dataset( 565 store, 566 mask_and_scale=mask_and_scale, 567 decode_times=decode_times, 568 concat_characters=concat_characters, 569 decode_coords=decode_coords, 570 drop_variables=drop_variables, 571 use_cftime=use_cftime, 572 decode_timedelta=decode_timedelta, 573 ) 574 return ds

File ~/.local/src/miniconda/envs/uptodatexarray/lib/python3.10/site-packages/xarray/backends/store.py:27, in StoreBackendEntrypoint.open_dataset(self, store, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta) 24 vars, attrs = store.load() 25 encoding = store.get_encoding() ---> 27 vars, attrs, coord_names = conventions.decode_cf_variables( 28 vars, 29 attrs, 30 mask_and_scale=mask_and_scale, 31 decode_times=decode_times, 32 concat_characters=concat_characters, 33 decode_coords=decode_coords, 34 drop_variables=drop_variables, 35 use_cftime=use_cftime, 36 decode_timedelta=decode_timedelta, 37 ) 39 ds = Dataset(vars, attrs=attrs) 40 ds = ds.set_coords(coord_names.intersection(vars))

File ~/.local/src/miniconda/envs/uptodatexarray/lib/python3.10/site-packages/xarray/conventions.py:503, in decode_cf_variables(variables, attributes, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta) 499 continue 500 stack_char_dim = ( 501 concat_characters and v.dtype == "S1" and v.ndim > 0 and stackable(v.dims[-1]) 502 ) --> 503 new_vars[k] = decode_cf_variable( 504 k, 505 v, 506 concat_characters=concat_characters, 507 mask_and_scale=mask_and_scale, 508 decode_times=decode_times, 509 stack_char_dim=stack_char_dim, 510 use_cftime=use_cftime, 511 decode_timedelta=decode_timedelta, 512 ) 513 if decode_coords in [True, "coordinates", "all"]: 514 var_attrs = new_vars[k].attrs

File ~/.local/src/miniconda/envs/uptodatexarray/lib/python3.10/site-packages/xarray/conventions.py:354, in decode_cf_variable(name, var, concat_characters, mask_and_scale, decode_times, decode_endianness, stack_char_dim, use_cftime, decode_timedelta) 351 var = coder.decode(var, name=name) 353 if decode_timedelta: --> 354 var = times.CFTimedeltaCoder().decode(var, name=name) 355 if decode_times: 356 var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name)

File ~/.local/src/miniconda/envs/uptodatexarray/lib/python3.10/site-packages/xarray/coding/times.py:537, in CFTimedeltaCoder.decode(self, variable, name) 534 def decode(self, variable, name=None): 535 dims, data, attrs, encoding = unpack_for_decoding(variable) --> 537 if "units" in attrs and attrs["units"] in TIME_UNITS: 538 units = pop_to(attrs, encoding, "units") 539 transform = partial(decode_cf_timedelta, units=units)

TypeError: unhashable type: 'numpy.ndarray' ```

Anything else we need to know?

The following assignation produces the bug :

python ds.temperature.attrs["units"] = []

But these ones does not produce the bug : python ds.temperature.attrs["units"] = "[]" ds.temperature.attrs["units"] = ""

Also, i don't know how the units attributes get encoded for writing but i see no difference between ds.temperature.attrs["units"] = "" and ds.temperature.attrs["units"] = [] when using ncdump on the file

Environment

This bug was encountered with versions below this one.

In the xarray.core.dataset.pymodule, more particulary in the reindex method the tolerance argument is set to be a Number

https://github.com/pydata/xarray/blob/f08672847abec18f46df75e2f620646d27fa41a2/xarray/core/dataset.py#L2743

But the _reindex function call reindex_variable function. In the reindex_variable the type for tolerance is Any. This function ends to call get_indexer_nd which call the pandas function get_indexer :

https://github.com/pydata/xarray/blob/f08672847abec18f46df75e2f620646d27fa41a2/xarray/core/indexes.py#L137

In pandas the type of tolerance according to the docs can be a scalar or a list-like object

• [x] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

When using xr.open_mfdataset() with two exact same files by mistake, it causes an unclear error message

What happened:

With of course the time dimension existing :

python ds=xr.open_mfdataset(["some_file.nc","some_file.nc"],concat_dim="time",engine="netcdf4")

```python

ValueError Traceback (most recent call last)

~/.local/src/miniconda/envs/minireobs/lib/python3.8/site-packages/xarray/backends/api.py in open_mfdataset(paths, chunks, concat_dim, compat, preprocess, engine, lock, data_vars, coords, combine, parallel, join, attrs_file, **kwargs) 966 # Redo ordering from coordinates, ignoring how they were ordered 967 # previously --> 968 combined = combine_by_coords( 969 datasets, 970 compat=compat,

~/.local/src/miniconda/envs/minireobs/lib/python3.8/site-packages/xarray/core/combine.py in combine_by_coords(datasets, compat, data_vars, coords, fill_value, join, combine_attrs) 762 concatenated_grouped_by_data_vars = [] 763 for vars, datasets_with_same_vars in grouped_by_vars: --> 764 combined_ids, concat_dims = _infer_concat_order_from_coords( 765 list(datasets_with_same_vars) 766 )

~/.local/src/miniconda/envs/minireobs/lib/python3.8/site-packages/xarray/core/combine.py in _infer_concat_order_from_coords(datasets) 106 107 if len(datasets) > 1 and not concat_dims: --> 108 raise ValueError( 109 "Could not find any dimension coordinates to use to " 110 "order the datasets for concatenation"

ValueError: Could not find any dimension coordinates to use to order the datasets for concatenation ``` What you expected to happen: A warning saying that we are using the same dataset ? A more explicit error message (exact same dimensions) ? No error and no concatenation, remove duplicated datasets?

Environment: