What is your issue?

In #8125 breaking tests in Upstream-CI was reported. Root cause is upstream #https://github.com/pandas-dev/pandas/issues/54920 which is tackled in https://github.com/pandas-dev/pandas/pull/54927.

To get the tests working, the fill_value-kwarg has been removed in #8139. The tests didn't cover the use of fill_value for scipy based interpolators (the used np.nan is the default-value) completely.

This issue is to keep track of adding proper tests for fill_value.

What happened?

When reading nanosecond precision time data from netcdf the precision is lost. This happens because CFMaskCoder will convert the variable to floating point and insert "NaN". In CFDatetimeCoder the floating point is cast back to int64 to transform into datetime64. This casting is sometimes undefined, hence #7098.

What did you expect to happen?

Precision should be preserved. The transformation to floating point should be omitted.

Minimal Complete Verifiable Example

```Python import xarray as xr import numpy as np import netCDF4 as nc import matplotlib.pyplot as plt

create time array and fillvalue

min_ns = -9223372036854775808 max_ns = 9223372036854775807 cnt = 2000 time_arr = np.arange(min_ns, min_ns + cnt, dtype=np.int64).astype("M8[ns]") fill_value = np.datetime64("1900-01-01", "ns")

create ncfile with time with attached _FillValue

with nc.Dataset("test.nc", mode="w") as ds: ds.createDimension("x", cnt) time = ds.createVariable("time", "<i8", ("x",), fill_value=fill_value) time[:] = time_arr time.units = "nanoseconds since 1970-01-01"

normal decoding

with xr.open_dataset("test.nc").load() as xr_ds: print("--- normal decoding ----------------------") print(xr_ds["time"]) plt.plot(xr_ds["time"].values.astype(np.int64) + max_ns, color="g", label="normal")

no decoding

with xr.open_dataset("test.nc", decode_cf=False).load() as xr_ds: print("--- no decoding ----------------------") print(xr_ds["time"]) plt.plot(xr_ds["time"].values + max_ns, lw=5, color="b", label="raw")

do not decode times, this shows how the CFMaskCoder converts

the array to floating point before it would run CFDatetimeCoder

with xr.open_dataset("test.nc", decode_times=False).load() as xr_ds: print("--- no time decoding ----------------------") print(xr_ds["time"])

do not run CFMaskCoder to show that times will be converted nicely

with CFDatetimeCoder

with xr.open_dataset("test.nc", mask_and_scale=False).load() as xr_ds: print("--- no masking ------------------------------") print(xr_ds["time"]) plt.plot(xr_ds["time"].values.astype(np.int64) + max_ns, lw=2, color="r", label="nomask")

plt.legend() ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

Python --- normal decoding ---------------------- <xarray.DataArray 'time' (x: 2000)> array([ 'NaT', 'NaT', 'NaT', ..., '1677-09-21T00:12:43.145226240', '1677-09-21T00:12:43.145226240', '1677-09-21T00:12:43.145226240'], dtype='datetime64[ns]') Dimensions without coordinates: x --- no decoding ---------------------- <xarray.DataArray 'time' (x: 2000)> array([-9223372036854775808, -9223372036854775807, -9223372036854775806, ..., -9223372036854773811, -9223372036854773810, -9223372036854773809]) Dimensions without coordinates: x Attributes: _FillValue: -2208988800000000000 units: nanoseconds since 1970-01-01 --- no time decoding ---------------------- <xarray.DataArray 'time' (x: 2000)> array([-9.22337204e+18, -9.22337204e+18, -9.22337204e+18, ..., -9.22337204e+18, -9.22337204e+18, -9.22337204e+18]) Dimensions without coordinates: x Attributes: units: nanoseconds since 1970-01-01 --- no masking ------------------------------ <xarray.DataArray 'time' (x: 2000)> array([ 'NaT', '1677-09-21T00:12:43.145224193', '1677-09-21T00:12:43.145224194', ..., '1677-09-21T00:12:43.145226189', '1677-09-21T00:12:43.145226190', '1677-09-21T00:12:43.145226191'], dtype='datetime64[ns]') Dimensions without coordinates: x Attributes: _FillValue: -2208988800000000000

Anything else we need to know?

Plot from above code:

Xref: #7098, https://github.com/pydata/xarray/issues/7790#issuecomment-1531050846

Environment

What happened?

When reading a netcdf file wich has been created with fill_value=None (default) those data is not masked. If one is writing back to disk this manifests.

What did you expect to happen?

Values should be masked.

There seems to be a simple solution:

On read apply the netcdf default fill_value in the variables attributes before decoding if no _FillValue attribute is set. After decoding we could change that to np.nan for floating point types.

Minimal Complete Verifiable Example

```Python import numpy as np import netCDF4 as nc import xarray as xr

with nc.Dataset("test-no-missing-01.nc", mode="w") as ds: x = ds.createDimension("x", 5) test = ds.createVariable("test", "f4", ("x",), fill_value=None) test[:4] = np.array([0.0, np.nan, 1.0, 8.0], dtype="f4") with nc.Dataset("test-no-missing-01.nc") as ds: print(ds["test"]) print(ds["test"][:])

with xr.open_dataset("test-no-missing-01.nc").load() as roundtrip: print(roundtrip) print(roundtrip["test"].attrs) print(roundtrip["test"].encoding) roundtrip.to_netcdf("test-no-missing-02.nc") with nc.Dataset("test-no-missing-02.nc") as ds: print(ds["test"]) print(ds["test"][:]) ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python <class 'netCDF4._netCDF4.Variable'> float32 test(x) unlimited dimensions: current shape = (5,) filling on, default _FillValue of 9.969209968386869e+36 used [0.0 nan 1.0 8.0 --]

<xarray.Dataset> Dimensions: (x: 5) Dimensions without coordinates: x Data variables: test (x) float32 0.0 nan 1.0 8.0 9.969e+36 {} {'zlib': False, 'szip': False, 'zstd': False, 'bzip2': False, 'blosc': False, 'shuffle': False, 'complevel': 0, 'fletcher32': False, 'contiguous': True, 'chunksizes': None, 'source': 'test-no-missing-01.nc', 'original_shape': (5,), 'dtype': dtype('float32')} <class 'netCDF4._netCDF4.Variable'> float32 test(x) _FillValue: nan unlimited dimensions: current shape = (5,) filling on [0.0 -- 1.0 8.0 9.969209968386869e+36] ```

Anything else we need to know?

The issue is similar to #7722 but is more intricate, as now the status of certain data values change from masked to some netcdf specific default value.

This is when only parts of the source dataset have been written to. Then the default fill_value get's delivered to the user but it is not backed by an _FillValue attribute.

Environment

What happened?

see also #7191

If missing_value and _FillValue is an attribute of a DataArray it can't be written out to file if these two contradict:

python ValueError: Variable 'test' has conflicting _FillValue (nan) and missing_value (1.0). Cannot encode data.

This happens, if missing_value is an attribute of a specific netCDF Dataset of an existing file. On read the missing_value will be masked with np.nan on the data and it will be preserved within encoding. On write, _FillValue will be added as attribute by xarray (if not available, at least for floating point types), too. So far so good.

The error first manifests if you read back this file and try to write it again. There is no warning on the second read, that the two _FillValue and missing_value are differing. Only on the second write.

What did you expect to happen?

The file should be written on the second roundtrip.

There are at least two solutions to this:

1. Mask missing_value on read and purge missing_value completely in favor of _FillValue.
2. Do not handle missing_value at all, but let the user take action.

Minimal Complete Verifiable Example

```Python import numpy as np import netCDF4 as nc import xarray as xr

with nc.Dataset("test-no-fillval-01.nc", mode="w") as ds: x = ds.createDimension("x", 4) test = ds.createVariable("test", "f4", ("x",), fill_value=None) test.missing_value = 1. test.valid_min = 2. test.valid_max = 10. test[:] = np.array([0.0, np.nan, 1.0, 8.0], dtype="f4") with nc.Dataset("test-no-fillval-01.nc") as ds: print(ds["test"]) print(ds["test"][:])

with xr.open_dataset("test-no-fillval-01.nc").load() as roundtrip: print(roundtrip) print(roundtrip["test"].attrs) print(roundtrip["test"].encoding) roundtrip.to_netcdf("test-no-fillval-02.nc")

with xr.open_dataset("test-no-fillval-02.nc").load() as roundtrip: print(roundtrip) print(roundtrip["test"].attrs) print(roundtrip["test"].encoding) roundtrip.to_netcdf("test-no-fillval-03.nc") ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python <class 'netCDF4._netCDF4.Variable'> float32 test(x) missing_value: 1.0 valid_min: 2.0 valid_max: 10.0 unlimited dimensions: current shape = (4,) filling on, default _FillValue of 9.969209968386869e+36 used

<xarray.Dataset> Dimensions: (x: 4) Dimensions without coordinates: x Data variables: test (x) float32 0.0 nan nan 8.0 {'valid_min': 2.0, 'valid_max': 10.0} {'zlib': False, 'szip': False, 'zstd': False, 'bzip2': False, 'blosc': False, 'shuffle': False, 'complevel': 0, 'fletcher32': False, 'contiguous': True, 'chunksizes': None, 'source': 'test-no-fillval-01.nc', 'original_shape': (4,), 'dtype': dtype('float32'), 'missing_value': 1.0}

<xarray.Dataset> Dimensions: (x: 4) Dimensions without coordinates: x Data variables: test (x) float32 0.0 nan nan 8.0 {'valid_min': 2.0, 'valid_max': 10.0} {'zlib': False, 'szip': False, 'zstd': False, 'bzip2': False, 'blosc': False, 'shuffle': False, 'complevel': 0, 'fletcher32': False, 'contiguous': True, 'chunksizes': None, 'source': 'test-no-fillval-02.nc', 'original_shape': (4,), 'dtype': dtype('float32'), 'missing_value': 1.0, '_FillValue': nan}

File /home/kai/miniconda/envs/xarray_311/lib/python3.11/site-packages/xarray/coding/variables.py:167, in CFMaskCoder.encode(self, variable, name) 160 mv = encoding.get("missing_value") 162 if ( 163 fv is not None 164 and mv is not None 165 and not duck_array_ops.allclose_or_equiv(fv, mv) 166 ): --> 167 raise ValueError( 168 f"Variable {name!r} has conflicting _FillValue ({fv}) and missing_value ({mv}). Cannot encode data." 169 ) 171 if fv is not None: 172 # Ensure _FillValue is cast to same dtype as data's 173 encoding["_FillValue"] = dtype.type(fv)

ValueError: Variable 'test' has conflicting _FillValue (nan) and missing_value (1.0). Cannot encode data. ```

Anything else we need to know?

The adding of _FillValue on write happens here:

https://github.com/pydata/xarray/blob/d4db16699f30ad1dc3e6861601247abf4ac96567/xarray/conventions.py#L300

https://github.com/pydata/xarray/blob/d4db16699f30ad1dc3e6861601247abf4ac96567/xarray/conventions.py#L144-L152

Environment

MCVE Code Sample

```python

Your code here

import netCDF4 import h5netcdf import xarray from xarray.backends.common import find_root_and_group root = netCDF4.Dataset("test.nc", "w", format="NETCDF4") test1 = root.createGroup("test1") test1.createGroup("test2") root.close() h5n = h5netcdf.File('test.nc') print(find_root_and_group(h5n['test1'])[1]) h5n.close() ```

This will output: //test1//test1/test2

Expected Output

/test1/test2

Problem Description

I stumbled over this while working on #3618. Although the function claims to retrieve root and group name of netCDF4/h5netcdf datasets the output for h5netcdf is broken.

Output of xr.show_versions()

What happened:

Decoding of times fails for NetCDF4 CfRadial2 style file due to time_reference-string is in another variable.

What you expected to happen:

time_reference should be extracted from the variable and time should be decoded properly.

Minimal Complete Verifiable Example:

```python

create file with such feature

time = np.arange(10) time_attrs = dict(standard_name="time", units="seconds since time_reference", calendar="gregorian") time_ref_attrs = dict(comments="UTC reference date. Format follows ISO 8601 standard.")

ds = xr.Dataset( data_vars=dict( time=(["time"], time, time_attrs),
time_reference=([], '1970-01-01T00:00:00Z', time_ref_attrs), ))

ds.to_netcdf("test.nc")

breaks with the error below

with xr.open_dataset("test.nc") as ds: print(ds)

ad-hoc fix

with xr.open_dataset("test.nc", decode_times=False) as ds: tr = ds.time.attrs["units"].split(" ") nt = ds[tr[-1]].item() ds.time.attrs["units"] = " ".join([*tr[:-1], nt]) ds = xr.decode_cf(ds) print(ds) ```

```python

AttributeError Traceback (most recent call last) ~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in _decode_cf_datetime_dtype(data, units, calendar, use_cftime) 142 try: --> 143 result = decode_cf_datetime(example_value, units, calendar, use_cftime) 144 except Exception:

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in decode_cf_datetime(num_dates, units, calendar, use_cftime) 224 try: --> 225 dates = _decode_datetime_with_pandas(flat_num_dates, units, calendar) 226 except (KeyError, OutOfBoundsDatetime, OverflowError):

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in _decode_datetime_with_pandas(flat_num_dates, units, calendar) 174 --> 175 delta, ref_date = _unpack_netcdf_time_units(units) 176 delta = _netcdf_to_numpy_timeunit(delta)

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in _unpack_netcdf_time_units(units) 127 delta_units, ref_date = [s.strip() for s in matches.groups()] --> 128 ref_date = _ensure_padded_year(ref_date) 129

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in _ensure_padded_year(ref_date) 102 matches_start_digits = re.match(r"(\d+)(.*)", ref_date) --> 103 ref_year, everything_else = [s for s in matches_start_digits.groups()] 104 ref_date_padded = "{:04d}{}".format(int(ref_year), everything_else)

AttributeError: 'NoneType' object has no attribute 'groups'

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last) <ipython-input-76-da468c6317bb> in <module> 13 ds.to_netcdf("test.nc") 14 ---> 15 with xr.open_dataset("test.nc") as ds: 16 print(ds) 17

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs, use_cftime, decode_timedelta) 432 from . import apiv2 433 --> 434 return apiv2.open_dataset(**kwargs) 435 436 if mask_and_scale is None:

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/backends/apiv2.py in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, backend_kwargs, **kwargs) 266 267 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None) --> 268 backend_ds = backend.open_dataset( 269 filename_or_obj, 270 drop_variables=drop_variables,

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/backends/netCDF4_.py in open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, format, clobber, diskless, persist, lock, autoclose) 557 store_entrypoint = StoreBackendEntrypoint() 558 with close_on_error(store): --> 559 ds = store_entrypoint.open_dataset( 560 store, 561 mask_and_scale=mask_and_scale,

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/backends/store.py in open_dataset(self, store, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta) 23 encoding = store.get_encoding() 24 ---> 25 vars, attrs, coord_names = conventions.decode_cf_variables( 26 vars, 27 attrs,

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/conventions.py in decode_cf_variables(variables, attributes, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta) 510 and stackable(v.dims[-1]) 511 ) --> 512 new_vars[k] = decode_cf_variable( 513 k, 514 v,

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/conventions.py in decode_cf_variable(name, var, concat_characters, mask_and_scale, decode_times, decode_endianness, stack_char_dim, use_cftime, decode_timedelta) 358 var = times.CFTimedeltaCoder().decode(var, name=name) 359 if decode_times: --> 360 var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name) 361 362 dimensions, data, attributes, encoding = variables.unpack_for_decoding(var)

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in decode(self, variable, name) 515 units = pop_to(attrs, encoding, "units") 516 calendar = pop_to(attrs, encoding, "calendar") --> 517 dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime) 518 transform = partial( 519 decode_cf_datetime,

~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in _decode_cf_datetime_dtype(data, units, calendar, use_cftime) 151 "if it is not installed." 152 ) --> 153 raise ValueError(msg) 154 else: 155 dtype = getattr(result, "dtype", np.dtype("object"))

ValueError: unable to decode time units 'seconds since time_reference' with "calendar 'gregorian'". Try opening your dataset with decode_times=False or installing cftime if it is not installed. ```

Anything else we need to know?:

I've searched the issues to no avail, also internet search wasn't successful so far. Any pointers welcome.

Environment:

Code Sample, a copy-pastable example if possible

A "Minimal, Complete and Verifiable Example" will make it much easier for maintainers to help you: http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

• Case 1: Creation of Dataset without Coordinates python data = np.zeros((2,3)) ds = xr.Dataset({'test': (['c', 'b'], data)}) print(ds.dims) ds2 = xr.concat([ds, ds], dim='c') print(ds2.dims) yields (assumed correct) output of: Frozen(SortedKeysDict({'c': 2, 'b': 3})) Frozen(SortedKeysDict({'c': 4, 'b': 3}))
• Case 2: Creation of Dataset with Coordinates python data = np.zeros((2,3)) ds = xr.Dataset({'test': (['c', 'b'], data)}, coords={'c': (['c'], np.arange(data.shape[0])), 'b': (['b'], np.arange(data.shape[1])),}) print(ds.dims) ds2 = xr.concat([ds, ds], dim='c') print(ds2.dims) yields (assumed false) output of: Frozen(SortedKeysDict({'c': 2, 'b': 3})) Frozen(SortedKeysDict({'b': 3, 'c': 4}))

Problem description

xr.concat changes the dimension order for .dims as well as .sizes to an alphanumerically sorted representation.

Expected Output

xr.concat should not change the dimension order in any case.

Frozen(SortedKeysDict({'c': 2, 'b': 3})) Frozen(SortedKeysDict({'c': 4, 'b': 3}))

Output of xr.show_versions()

What happened:

When using xr.open_dataset with H5NetCDFDataStore and opened h5py.File handle deepcopy in Dataset.sortby/align leads to TypeError: h5py objects cannot be pickled.

What you expected to happen:

While applying Dataset.sortby no error should be raised.

Minimal Complete Verifiable Example:

```python

create hdf5 file

import h5py f = h5py.File('myfile.h5','w') dset = f.create_dataset("data", (360, 1000)) f.close()

import h5netcdf import xarray as xr import numpy as np

f = h5netcdf.File("myfile.h5", "r", phony_dims="access") s0 = xr.backends.H5NetCDFStore(f) ds = xr.open_dataset(s0, engine="h5netcdf", chunks=None) ds = ds.assign_coords({"phony_dim_0": np.arange(ds.dims['phony_dim_0'], 0, -1)}) ds.sortby('phony_dim_0') ds.close() ```

Anything else we need to know?:

When invoked with chunks={} it works as well as if the following code is used:

python ds = xr.open_dataset('myfile.h5', group='/', engine='h5netcdf', backend_kwargs=dict(phony_dims='access')) ds = ds.assign_coords({"phony_dim_0": np.arange(ds.dims['phony_dim_0'], 0, -1)}) ds.sortby('phony_dim_0') ds.close()

This was introduced by #4221, see https://github.com/pydata/xarray/blob/66ab0ae4f3aa3c461357a5a895405e81357796b1/xarray/core/variable.py#L939-L941

Before:

python if deep and ( hasattr(data, "__array_function__") or isinstance(data, dask_array_type) or (not IS_NEP18_ACTIVE and isinstance(data, np.ndarray)) ): data = copy.deepcopy(data)

All three of the above tests return False in my case, so deepcopy should never be used here.

Environment: