Aims to resolve parts of #8844.

python xarray/tests/test_accessor_str.py::test_case_str: AssertionError: assert dtype('<U26') == dtype('<U30') I'm not sure this is the right location for the fix, at least it fixes those errors. AFAICT this is some issue somewhere inside apply_ufunc where the string dtype size is kept. So this fix removes the size information from the dtype (actually recreating it).

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] Closes #5563
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

This fixes two issues with numpy 2.0 (see #8844)

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [x] Closes #7691
• [x] Closes #5597
• [x] Closes #2304
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

This resurrects some of #7654. It takes special care of correctly handling dtypes when encoding/decoding packed data

That's a follow-up to #8571.

• [x] Closes #8442
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• reverse index output of bottleneck move_argmax/move_argmin functions
• add move_argmax/move_argmin to bottleneck tests

• [x] Closes #8541
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• [ ] Closes #xxxx
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

This is adding conformance tests for scale_factor/add_offset/dtype when encoding/decoding cf packed data with https://github.com/pydata/xarray/blob/c75ac8b7ab33be95f74d9e6f10b8173c68828751/xarray/coding/variables.py#L217. This doesn't include tests for _FillValue/missing_value/valid_range etc. which need a separate test.

The idea behind is two-fold:

• be strict when encoding to only write CF conforming data, raise Errors if non-conforming
• be liberal when decoding, but issue warnings

This is another step to move forward to finally fix issues like #2304 and a bunch more.

• [x] xref #7652
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst
• includes
  • implement coders
  • preserve boolean-dtype within encoding, adapt test
  • determine cf packed data type
  • ~transform numpy object-dtype strings (vlen) to numpy unicode strings~

This should also fix:

• Closes https://github.com/pydata/xarray/issues/7691
• Closes https://github.com/pydata/xarray/issues/2304 and possible other issues which have the float32/float64 issue

• [x] Closes (partly) #1647
• [x] Closes #7652
• [x] Closes #7868
• [x] Tests adapted
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] Closes #8269
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

As requested in #8223 I'm preparing for Xarray release 2023.09.0.

ToDo:

• [x] Add final date
• [x] Approve headline and other changes in whats-new.rst
• [x] add plan-to-merge PR

• [x] Closes #8223
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

• decode variable with mismatched coordinate attribute
• propagate coordinates to encoding in any case

• [x] Closes #1809
• [x] Closes #5170
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Update: The enhanced test works well in Upstream-dev tests, Pandas 2.1.1 is available for safe merge.

• [x] Closes #8146
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

What is your issue?

In #8125 breaking tests in Upstream-CI was reported. Root cause is upstream #https://github.com/pandas-dev/pandas/issues/54920 which is tackled in https://github.com/pandas-dev/pandas/pull/54927.

To get the tests working, the fill_value-kwarg has been removed in #8139. The tests didn't cover the use of fill_value for scipy based interpolators (the used np.nan is the default-value) completely.

This issue is to keep track of adding proper tests for fill_value.

• [x] Closes #1064
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] Closes #6570

• [x] Closes #7817,
• [x] closes #7942,
• [x] closes #7790,
• [x] closes #6191
• [x] closes #7096
• [x] closes #1064
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Closes #7098 as not needed, but worked around to preserve the fast int-based timedelta calculation.

What happened?

When reading nanosecond precision time data from netcdf the precision is lost. This happens because CFMaskCoder will convert the variable to floating point and insert "NaN". In CFDatetimeCoder the floating point is cast back to int64 to transform into datetime64. This casting is sometimes undefined, hence #7098.

What did you expect to happen?

Precision should be preserved. The transformation to floating point should be omitted.

Minimal Complete Verifiable Example

```Python import xarray as xr import numpy as np import netCDF4 as nc import matplotlib.pyplot as plt

create time array and fillvalue

min_ns = -9223372036854775808 max_ns = 9223372036854775807 cnt = 2000 time_arr = np.arange(min_ns, min_ns + cnt, dtype=np.int64).astype("M8[ns]") fill_value = np.datetime64("1900-01-01", "ns")

create ncfile with time with attached _FillValue

with nc.Dataset("test.nc", mode="w") as ds: ds.createDimension("x", cnt) time = ds.createVariable("time", "<i8", ("x",), fill_value=fill_value) time[:] = time_arr time.units = "nanoseconds since 1970-01-01"

normal decoding

with xr.open_dataset("test.nc").load() as xr_ds: print("--- normal decoding ----------------------") print(xr_ds["time"]) plt.plot(xr_ds["time"].values.astype(np.int64) + max_ns, color="g", label="normal")

no decoding

with xr.open_dataset("test.nc", decode_cf=False).load() as xr_ds: print("--- no decoding ----------------------") print(xr_ds["time"]) plt.plot(xr_ds["time"].values + max_ns, lw=5, color="b", label="raw")

do not decode times, this shows how the CFMaskCoder converts

the array to floating point before it would run CFDatetimeCoder

with xr.open_dataset("test.nc", decode_times=False).load() as xr_ds: print("--- no time decoding ----------------------") print(xr_ds["time"])

do not run CFMaskCoder to show that times will be converted nicely

with CFDatetimeCoder

with xr.open_dataset("test.nc", mask_and_scale=False).load() as xr_ds: print("--- no masking ------------------------------") print(xr_ds["time"]) plt.plot(xr_ds["time"].values.astype(np.int64) + max_ns, lw=2, color="r", label="nomask")

plt.legend() ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

Python --- normal decoding ---------------------- <xarray.DataArray 'time' (x: 2000)> array([ 'NaT', 'NaT', 'NaT', ..., '1677-09-21T00:12:43.145226240', '1677-09-21T00:12:43.145226240', '1677-09-21T00:12:43.145226240'], dtype='datetime64[ns]') Dimensions without coordinates: x --- no decoding ---------------------- <xarray.DataArray 'time' (x: 2000)> array([-9223372036854775808, -9223372036854775807, -9223372036854775806, ..., -9223372036854773811, -9223372036854773810, -9223372036854773809]) Dimensions without coordinates: x Attributes: _FillValue: -2208988800000000000 units: nanoseconds since 1970-01-01 --- no time decoding ---------------------- <xarray.DataArray 'time' (x: 2000)> array([-9.22337204e+18, -9.22337204e+18, -9.22337204e+18, ..., -9.22337204e+18, -9.22337204e+18, -9.22337204e+18]) Dimensions without coordinates: x Attributes: units: nanoseconds since 1970-01-01 --- no masking ------------------------------ <xarray.DataArray 'time' (x: 2000)> array([ 'NaT', '1677-09-21T00:12:43.145224193', '1677-09-21T00:12:43.145224194', ..., '1677-09-21T00:12:43.145226189', '1677-09-21T00:12:43.145226190', '1677-09-21T00:12:43.145226191'], dtype='datetime64[ns]') Dimensions without coordinates: x Attributes: _FillValue: -2208988800000000000

Anything else we need to know?

Plot from above code:

Xref: #7098, https://github.com/pydata/xarray/issues/7790#issuecomment-1531050846

Environment

While working on #8146 this came up. Aligning with upstream naming convention.

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] Closes #7928
• [ ] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

What happened?

When reading a netcdf file wich has been created with fill_value=None (default) those data is not masked. If one is writing back to disk this manifests.

What did you expect to happen?

Values should be masked.

There seems to be a simple solution:

On read apply the netcdf default fill_value in the variables attributes before decoding if no _FillValue attribute is set. After decoding we could change that to np.nan for floating point types.

Minimal Complete Verifiable Example

```Python import numpy as np import netCDF4 as nc import xarray as xr

with nc.Dataset("test-no-missing-01.nc", mode="w") as ds: x = ds.createDimension("x", 5) test = ds.createVariable("test", "f4", ("x",), fill_value=None) test[:4] = np.array([0.0, np.nan, 1.0, 8.0], dtype="f4") with nc.Dataset("test-no-missing-01.nc") as ds: print(ds["test"]) print(ds["test"][:])

with xr.open_dataset("test-no-missing-01.nc").load() as roundtrip: print(roundtrip) print(roundtrip["test"].attrs) print(roundtrip["test"].encoding) roundtrip.to_netcdf("test-no-missing-02.nc") with nc.Dataset("test-no-missing-02.nc") as ds: print(ds["test"]) print(ds["test"][:]) ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python <class 'netCDF4._netCDF4.Variable'> float32 test(x) unlimited dimensions: current shape = (5,) filling on, default _FillValue of 9.969209968386869e+36 used [0.0 nan 1.0 8.0 --]

<xarray.Dataset> Dimensions: (x: 5) Dimensions without coordinates: x Data variables: test (x) float32 0.0 nan 1.0 8.0 9.969e+36 {} {'zlib': False, 'szip': False, 'zstd': False, 'bzip2': False, 'blosc': False, 'shuffle': False, 'complevel': 0, 'fletcher32': False, 'contiguous': True, 'chunksizes': None, 'source': 'test-no-missing-01.nc', 'original_shape': (5,), 'dtype': dtype('float32')} <class 'netCDF4._netCDF4.Variable'> float32 test(x) _FillValue: nan unlimited dimensions: current shape = (5,) filling on [0.0 -- 1.0 8.0 9.969209968386869e+36] ```

Anything else we need to know?

The issue is similar to #7722 but is more intricate, as now the status of certain data values change from masked to some netcdf specific default value.

This is when only parts of the source dataset have been written to. Then the default fill_value get's delivered to the user but it is not backed by an _FillValue attribute.

Environment

• [x] Closes #7943
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

What happened?

see also #7191

If missing_value and _FillValue is an attribute of a DataArray it can't be written out to file if these two contradict:

python ValueError: Variable 'test' has conflicting _FillValue (nan) and missing_value (1.0). Cannot encode data.

This happens, if missing_value is an attribute of a specific netCDF Dataset of an existing file. On read the missing_value will be masked with np.nan on the data and it will be preserved within encoding. On write, _FillValue will be added as attribute by xarray (if not available, at least for floating point types), too. So far so good.

The error first manifests if you read back this file and try to write it again. There is no warning on the second read, that the two _FillValue and missing_value are differing. Only on the second write.

What did you expect to happen?

The file should be written on the second roundtrip.

There are at least two solutions to this:

1. Mask missing_value on read and purge missing_value completely in favor of _FillValue.
2. Do not handle missing_value at all, but let the user take action.

Minimal Complete Verifiable Example

```Python import numpy as np import netCDF4 as nc import xarray as xr

with nc.Dataset("test-no-fillval-01.nc", mode="w") as ds: x = ds.createDimension("x", 4) test = ds.createVariable("test", "f4", ("x",), fill_value=None) test.missing_value = 1. test.valid_min = 2. test.valid_max = 10. test[:] = np.array([0.0, np.nan, 1.0, 8.0], dtype="f4") with nc.Dataset("test-no-fillval-01.nc") as ds: print(ds["test"]) print(ds["test"][:])

with xr.open_dataset("test-no-fillval-01.nc").load() as roundtrip: print(roundtrip) print(roundtrip["test"].attrs) print(roundtrip["test"].encoding) roundtrip.to_netcdf("test-no-fillval-02.nc")

with xr.open_dataset("test-no-fillval-02.nc").load() as roundtrip: print(roundtrip) print(roundtrip["test"].attrs) print(roundtrip["test"].encoding) roundtrip.to_netcdf("test-no-fillval-03.nc") ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python <class 'netCDF4._netCDF4.Variable'> float32 test(x) missing_value: 1.0 valid_min: 2.0 valid_max: 10.0 unlimited dimensions: current shape = (4,) filling on, default _FillValue of 9.969209968386869e+36 used

<xarray.Dataset> Dimensions: (x: 4) Dimensions without coordinates: x Data variables: test (x) float32 0.0 nan nan 8.0 {'valid_min': 2.0, 'valid_max': 10.0} {'zlib': False, 'szip': False, 'zstd': False, 'bzip2': False, 'blosc': False, 'shuffle': False, 'complevel': 0, 'fletcher32': False, 'contiguous': True, 'chunksizes': None, 'source': 'test-no-fillval-01.nc', 'original_shape': (4,), 'dtype': dtype('float32'), 'missing_value': 1.0}

<xarray.Dataset> Dimensions: (x: 4) Dimensions without coordinates: x Data variables: test (x) float32 0.0 nan nan 8.0 {'valid_min': 2.0, 'valid_max': 10.0} {'zlib': False, 'szip': False, 'zstd': False, 'bzip2': False, 'blosc': False, 'shuffle': False, 'complevel': 0, 'fletcher32': False, 'contiguous': True, 'chunksizes': None, 'source': 'test-no-fillval-02.nc', 'original_shape': (4,), 'dtype': dtype('float32'), 'missing_value': 1.0, '_FillValue': nan}

File /home/kai/miniconda/envs/xarray_311/lib/python3.11/site-packages/xarray/coding/variables.py:167, in CFMaskCoder.encode(self, variable, name) 160 mv = encoding.get("missing_value") 162 if ( 163 fv is not None 164 and mv is not None 165 and not duck_array_ops.allclose_or_equiv(fv, mv) 166 ): --> 167 raise ValueError( 168 f"Variable {name!r} has conflicting _FillValue ({fv}) and missing_value ({mv}). Cannot encode data." 169 ) 171 if fv is not None: 172 # Ensure _FillValue is cast to same dtype as data's 173 encoding["_FillValue"] = dtype.type(fv)

ValueError: Variable 'test' has conflicting _FillValue (nan) and missing_value (1.0). Cannot encode data. ```

Anything else we need to know?

The adding of _FillValue on write happens here:

https://github.com/pydata/xarray/blob/d4db16699f30ad1dc3e6861601247abf4ac96567/xarray/conventions.py#L300

https://github.com/pydata/xarray/blob/d4db16699f30ad1dc3e6861601247abf4ac96567/xarray/conventions.py#L144-L152

Environment

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] Fixes downstream https://github.com/pangeo-data/xESMF/issues/36
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Fixes dask handling for implementation in #3871.

• [ ] Closes #xxxx
• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [x] Closes #508,
• [x] Closes #3545
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

This is another attempt to solve #508. Took inspiration from #3545 by @scottcha.

This follows along @dcherian's comment in the same above PR (https://github.com/pydata/xarray/pull/3545#issuecomment-586346637).

Update:

After review the variable order is estimated by order of appearance in the list of datasets. That keeps full backwards compatibility and is deterministic. Thanks @dcherian and @keewis for the suggestions.

• [x] Closes #4826
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

Split out of #7654. Follow up to #7719. Needs rebase after #7719 is merged.

This is a fix for #4927.

• time_reference is overwritten with the actual time string inside time_reference before decoding, but only if a variable with the name exists.
• time_reference name is kept in encoding (the name might change, eg. reference_time).
• time_reference is added back to units attribute when encoding, to make roundtrips possible.

• [x] Closes #4927
• [x] Tests added
• [ ] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• set decode_vlen_strings=True for h5netcdf backend
• unpin h5py

This is an attempt to align with h5py=>3.0.0 string decoding changes. Now all strings are read as bytes objects for variable-length strings, or numpy bytes arrays ('S' dtypes) for fixed-length strings [1]. From h5netcdf=0.10.0 kwarg decode_vlen_strings is available. This PR makes use of this to keep backwards compatibility with h5py=2 and conformance with netcdf4-python.

[1] https://docs.h5py.org/en/stable/strings.html

• [x] Closes #4570
• [ ] Tests added
• [ ] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

use dask.array.apply_gufunc in xr.apply_ufunc for multiple outputs when dask='parallelized', add/fix tests

• [x] Closes #1815, closes #4015
• [x] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Remaining Issues:

• [ ] fitting name for current dask_gufunc_kwargs
• [ ] rephrase dask docs to fit new behaviour
• [ ] combine output_core_dims and output_sizes, eg. xr.apply_ufunc(..., output_core_dims=[{"abc": 2]])

…ternal use of apply_ufunc (follow-up to #4060, fixes #4385)

• [x] Closes #4385
• [ ] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

follow-up on #4060

• [x] Closes #4372
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8

This adds support for the new phony_dims keyword for the h5netcdf backend which is available from h5netcdf version 0.8.0.

• [ ] Closes #xxxx
• [ ] Tests added
• [x] Passes isort -rc . && black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

This PR implements remaining functionality in encode_cf_variable/decode_cf_variable as VariableCoders (as left as todo by @shoyer). It adapts the needed tests and adds some typing (thanks @Illviljan). Otherwise there are no changes in function to code.

It splits out of #7654 which turns out to be hard to follow (at least for me).

The problem is that the result_vars-dict is filled in two separate loops, handling variables and coordinates (AFAICT). This can change the ordering of dimensions and coordinates depending over which dimensions is concatenated.

This fix hooks into the second loop and reinsert the variable in question at the appropriate position by popping it out of the result_vars. There might be a more elegant solution to this.

• [x] Closes #2811, closes #4072
• [x] Tests added
• [x] Passes isort . && black . && mypy . && flake8
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• fix regression in backend.plugins.remove_duplicates()
• switch pkg_resources -> importlib in tests/test_plugins.py

• [x] Closes #5944
• [x] Tests fixed (switch pkg_resources -> importlib)
• [ ] Tests added
• [x] Passes pre-commit run --all-files
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] Closes #6197

MCVE Code Sample

```python

Your code here

import netCDF4 import h5netcdf import xarray from xarray.backends.common import find_root_and_group root = netCDF4.Dataset("test.nc", "w", format="NETCDF4") test1 = root.createGroup("test1") test1.createGroup("test2") root.close() h5n = h5netcdf.File('test.nc') print(find_root_and_group(h5n['test1'])[1]) h5n.close() ```

This will output: //test1//test1/test2

Expected Output

/test1/test2

Problem Description

I stumbled over this while working on #3618. Although the function claims to retrieve root and group name of netCDF4/h5netcdf datasets the output for h5netcdf is broken.

Output of xr.show_versions()

What happened:

Decoding of times fails for NetCDF4 CfRadial2 style file due to time_reference-string is in another variable.

What you expected to happen:

time_reference should be extracted from the variable and time should be decoded properly.

Minimal Complete Verifiable Example:

```python

create file with such feature

time = np.arange(10) time_attrs = dict(standard_name="time", units="seconds since time_reference", calendar="gregorian") time_ref_attrs = dict(comments="UTC reference date. Format follows ISO 8601 standard.")

ds = xr.Dataset( data_vars=dict( time=(["time"], time, time_attrs),
time_reference=([], '1970-01-01T00:00:00Z', time_ref_attrs), ))

ds.to_netcdf("test.nc")

breaks with the error below

with xr.open_dataset("test.nc") as ds: print(ds)

ad-hoc fix

with xr.open_dataset("test.nc", decode_times=False) as ds: tr = ds.time.attrs["units"].split(" ") nt = ds[tr[-1]].item() ds.time.attrs["units"] = " ".join([*tr[:-1], nt]) ds = xr.decode_cf(ds) print(ds) ```

```python

AttributeError Traceback (most recent call last) ~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in _decode_cf_datetime_dtype(data, units, calendar, use_cftime) 142 try: --> 143 result = decode_cf_datetime(example_value, units, calendar, use_cftime) 144 except Exception:

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in decode_cf_datetime(num_dates, units, calendar, use_cftime) 224 try: --> 225 dates = _decode_datetime_with_pandas(flat_num_dates, units, calendar) 226 except (KeyError, OutOfBoundsDatetime, OverflowError):

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in _decode_datetime_with_pandas(flat_num_dates, units, calendar) 174 --> 175 delta, ref_date = _unpack_netcdf_time_units(units) 176 delta = _netcdf_to_numpy_timeunit(delta)

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in _unpack_netcdf_time_units(units) 127 delta_units, ref_date = [s.strip() for s in matches.groups()] --> 128 ref_date = _ensure_padded_year(ref_date) 129

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in _ensure_padded_year(ref_date) 102 matches_start_digits = re.match(r"(\d+)(.*)", ref_date) --> 103 ref_year, everything_else = [s for s in matches_start_digits.groups()] 104 ref_date_padded = "{:04d}{}".format(int(ref_year), everything_else)

AttributeError: 'NoneType' object has no attribute 'groups'

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last) <ipython-input-76-da468c6317bb> in <module> 13 ds.to_netcdf("test.nc") 14 ---> 15 with xr.open_dataset("test.nc") as ds: 16 print(ds) 17

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs, use_cftime, decode_timedelta) 432 from . import apiv2 433 --> 434 return apiv2.open_dataset(**kwargs) 435 436 if mask_and_scale is None:

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/backends/apiv2.py in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, backend_kwargs, **kwargs) 266 267 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None) --> 268 backend_ds = backend.open_dataset( 269 filename_or_obj, 270 drop_variables=drop_variables,

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/backends/netCDF4_.py in open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, format, clobber, diskless, persist, lock, autoclose) 557 store_entrypoint = StoreBackendEntrypoint() 558 with close_on_error(store): --> 559 ds = store_entrypoint.open_dataset( 560 store, 561 mask_and_scale=mask_and_scale,

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/backends/store.py in open_dataset(self, store, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta) 23 encoding = store.get_encoding() 24 ---> 25 vars, attrs, coord_names = conventions.decode_cf_variables( 26 vars, 27 attrs,

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/conventions.py in decode_cf_variables(variables, attributes, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta) 510 and stackable(v.dims[-1]) 511 ) --> 512 new_vars[k] = decode_cf_variable( 513 k, 514 v,

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/conventions.py in decode_cf_variable(name, var, concat_characters, mask_and_scale, decode_times, decode_endianness, stack_char_dim, use_cftime, decode_timedelta) 358 var = times.CFTimedeltaCoder().decode(var, name=name) 359 if decode_times: --> 360 var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name) 361 362 dimensions, data, attributes, encoding = variables.unpack_for_decoding(var)

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in decode(self, variable, name) 515 units = pop_to(attrs, encoding, "units") 516 calendar = pop_to(attrs, encoding, "calendar") --> 517 dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime) 518 transform = partial( 519 decode_cf_datetime,

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in _decode_cf_datetime_dtype(data, units, calendar, use_cftime) 151 "if it is not installed." 152 ) --> 153 raise ValueError(msg) 154 else: 155 dtype = getattr(result, "dtype", np.dtype("object"))

ValueError: unable to decode time units 'seconds since time_reference' with "calendar 'gregorian'". Try opening your dataset with decode_times=False or installing cftime if it is not installed. ```

Anything else we need to know?:

I've searched the issues to no avail, also internet search wasn't successful so far. Any pointers welcome.

Environment:

Code Sample, a copy-pastable example if possible

A "Minimal, Complete and Verifiable Example" will make it much easier for maintainers to help you: http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

• Case 1: Creation of Dataset without Coordinates python data = np.zeros((2,3)) ds = xr.Dataset({'test': (['c', 'b'], data)}) print(ds.dims) ds2 = xr.concat([ds, ds], dim='c') print(ds2.dims) yields (assumed correct) output of: Frozen(SortedKeysDict({'c': 2, 'b': 3})) Frozen(SortedKeysDict({'c': 4, 'b': 3}))
• Case 2: Creation of Dataset with Coordinates python data = np.zeros((2,3)) ds = xr.Dataset({'test': (['c', 'b'], data)}, coords={'c': (['c'], np.arange(data.shape[0])), 'b': (['b'], np.arange(data.shape[1])),}) print(ds.dims) ds2 = xr.concat([ds, ds], dim='c') print(ds2.dims) yields (assumed false) output of: Frozen(SortedKeysDict({'c': 2, 'b': 3})) Frozen(SortedKeysDict({'b': 3, 'c': 4}))

Problem description

xr.concat changes the dimension order for .dims as well as .sizes to an alphanumerically sorted representation.

Expected Output

xr.concat should not change the dimension order in any case.

Frozen(SortedKeysDict({'c': 2, 'b': 3})) Frozen(SortedKeysDict({'c': 4, 'b': 3}))

Output of xr.show_versions()

What happened:

When using xr.open_dataset with H5NetCDFDataStore and opened h5py.File handle deepcopy in Dataset.sortby/align leads to TypeError: h5py objects cannot be pickled.

What you expected to happen:

While applying Dataset.sortby no error should be raised.

Minimal Complete Verifiable Example:

```python

create hdf5 file

import h5py f = h5py.File('myfile.h5','w') dset = f.create_dataset("data", (360, 1000)) f.close()

import h5netcdf import xarray as xr import numpy as np

f = h5netcdf.File("myfile.h5", "r", phony_dims="access") s0 = xr.backends.H5NetCDFStore(f) ds = xr.open_dataset(s0, engine="h5netcdf", chunks=None) ds = ds.assign_coords({"phony_dim_0": np.arange(ds.dims['phony_dim_0'], 0, -1)}) ds.sortby('phony_dim_0') ds.close() ```

Anything else we need to know?:

When invoked with chunks={} it works as well as if the following code is used:

python ds = xr.open_dataset('myfile.h5', group='/', engine='h5netcdf', backend_kwargs=dict(phony_dims='access')) ds = ds.assign_coords({"phony_dim_0": np.arange(ds.dims['phony_dim_0'], 0, -1)}) ds.sortby('phony_dim_0') ds.close()

This was introduced by #4221, see https://github.com/pydata/xarray/blob/66ab0ae4f3aa3c461357a5a895405e81357796b1/xarray/core/variable.py#L939-L941

Before:

python if deep and ( hasattr(data, "__array_function__") or isinstance(data, dask_array_type) or (not IS_NEP18_ACTIVE and isinstance(data, np.ndarray)) ): data = copy.deepcopy(data)

All three of the above tests return False in my case, so deepcopy should never be used here.

Environment:

enable H5NetCDFStore to work with open h5netcdf.File and h5netcdf.Group objects, add test

• [x] Tests added
• [x] Passes black . && mypy . && flake8
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API