issue_comments

, the PR seems to solve my specific issue without changing the encoding

Great, thanks for testing.

@euronion There is a potential fix for your issue in #7654. It would be great, if you could have a closer look and test against that PR.

MCVE:

python fname = "test-7691.nc" import netCDF4 as nc with nc.Dataset(fname, "w") as ds0: ds0.createDimension("t", 5) ds0.createVariable("x", "int16", ("t",), fill_value=-32767) v = ds0.variables["x"] v.set_auto_maskandscale(False) v.add_offset = 278.297319296597 v.scale_factor = 1.16753614203674e-05 v[:] = np.array([-32768, -32767, -32766, 32767, 0]) with nc.Dataset(fname) as ds1: x1 = ds1["x"][:] print("netCDF4-python:", x1.dtype, x1) with xr.open_dataset(fname) as ds2: x2 = ds2["x"].values ds2.to_netcdf("test-7691-01.nc") print("xarray first read:", x2.dtype, x2) with xr.open_dataset("test-7691-01.nc") as ds3: x3 = ds3["x"].values print("xarray roundtrip:", x3.dtype, x3)

python netCDF4-python: float64 [277.9147410535744 -- 277.9147644042972 278.67988586425815 278.297319296597] xarray first read: float32 [277.91476 nan 277.91476 278.6799 278.29733] xarray roundtrip: float32 [ nan nan nan 278.6799 278.29733] I've confirmed that correctly promoting to float64 in CFMaskCoder solves this issue.

As this doesn't surface that often it might just happen here by accident. If the _FillValue/missing_value would be -32768 then the issue would not manifest.

So for NetCDF the default fillvalue for NC_SHORT (int16) is -32767. That means the promotion to float32 instead the needed float64 is the problem here (floating point precision).

Thanks for all the details, @euronion.

From what I can tell, everything is OK with the original file. It's using packed data: https://docs.unidata.ucar.edu/nug/current/best_practices.html#bp_Packed-Data-Values. The only thing what might be a bit off is why they didn't choose -32768 as _FillValue

As both scale_factor and add_offset are of dtype float64 in the original file the data should be unpacked to float64 according to NetCDF-specs.

The reason why this isn't done is because the

https://github.com/pydata/xarray/blob/020b4c07047189c5c788eca9e6e77d64b8989d58/xarray/conventions.py#L379-L384

CFMaskCoder will promote int16 to float32 unconditionally. This happens in dtypes.maybe_promote():

https://github.com/pydata/xarray/blob/020b4c07047189c5c788eca9e6e77d64b8989d58/xarray/core/dtypes.py#L67-L70

The CFScaleOffsetCoder itself is able to correctly convert this to the wanted dtype float64:

https://github.com/pydata/xarray/blob/e79eaf5acdcda62f27ce81f08e7e71839887d3d1/xarray/coding/variables.py#L235-L251

As this doesn't surface that often it might just happen here by accident. If the _FillValue/missing_value would be -32768 then the issue would not manifest.

Update: corrected to maybe_promote()