issues: 1136315478

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
1136315478 I_kwDOAMm_X85DusxW 6272 ds.to_netcdf() changes values of variable 2042458 closed 0     12 2022-02-13T19:15:20Z 2024-03-25T14:56:51Z 2024-03-25T14:56:50Z NONE      

What happened?

I am very puzzled by an odd behavior of ds.to_netcdf() that modifies the value of a variable by offseting partly the vlue by 44500.

I have list of file containing meteorological variables organized in three dimensions 'time', 'longitude', latitude'. The files are loaded using xr.open_mfdataset('*.nc'). No problem here. The dataset is loaded in chuncks using Dask.

In [11]: ds Out[11]: <xarray.Dataset> Dimensions: (longitude: 10, latitude: 7, level: 14, time: 35760) Coordinates: * longitude (longitude) float32 6.5 6.75 7.0 7.25 7.5 7.75 8.0 8.25 8.5 8.75 * latitude (latitude) float32 61.25 61.0 60.75 60.5 60.25 60.0 59.75 * level (level) int32 600 650 700 750 775 800 ... 900 925 950 975 1000 * time (time) datetime64[ns] 1978-10-01 ... 2021-08-31T23:00:00 Data variables: z (time, level, latitude, longitude) float32 dask.array<chunksize=(744, 14, 7, 10), meta=np.ndarray> t (time, level, latitude, longitude) float32 dask.array<chunksize=(744, 14, 7, 10), meta=np.ndarray> u (time, level, latitude, longitude) float32 dask.array<chunksize=(744, 14, 7, 10), meta=np.ndarray> v (time, level, latitude, longitude) float32 dask.array<chunksize=(744, 14, 7, 10), meta=np.ndarray> r (time, level, latitude, longitude) float32 dask.array<chunksize=(744, 14, 7, 10), meta=np.ndarray> q (time, level, latitude, longitude) float32 dask.array<chunksize=(744, 14, 7, 10), meta=np.ndarray> Attributes: Conventions: CF-1.6 history: 2021-12-08 19:22:33 GMT by grib_to_netcdf-2.23.0: /opt/ecmw...

Now, I would like to save a subset of this dataset to a netcdf file as follow: ds.isel(latitude[1,2,3], longitude=[3,4,5]).to_netcdf('sub.nc'). So far nothing particular.

The value z is a float32 which varies from 2000 to -2000 along the time dimension. After being saved in the subsample, z is still a float32 but the values that are less than -1000 are being offset by 44500. However, if I do (ds.z.isel(latitude[1,2,3], longitude=[3,4,5])*1).to_netcdf('sub.nc'), instead, then all values in the subsampled netcdf are fine. I am very puzzled by this behavior. Could this be an odd behavior of dask chuncks and to_netcdf()?

What did you expect to happen?

I expected no modification of the data after saving to netcdf, no matter what.

Minimal Complete Verifiable Example

I will share files upon request.

Relevant log output

No response

Anything else we need to know?

No response

Environment

INSTALLED VERSIONS

commit: None python: 3.9.7 (default, Sep 16 2021, 13:09:58) [GCC 7.5.0] python-bits: 64 OS: Linux OS-release: 5.13.0-28-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.10.6 libnetcdf: 4.8.1

xarray: 0.20.1 pandas: 1.3.5 numpy: 1.20.3 scipy: 1.7.3 netCDF4: 1.5.7 pydap: None h5netcdf: 0.11.0 h5py: 3.6.0 Nio: None zarr: None cftime: 1.5.1.1 nc_time_axis: None PseudoNetCDF: None rasterio: 1.2.8 cfgrib: None iris: None bottleneck: 1.3.2 dask: 2021.10.0 distributed: 2021.10.0 matplotlib: 3.5.0 cartopy: None seaborn: 0.11.2 numbagg: None fsspec: 2022.01.0 cupy: None pint: None sparse: None setuptools: 58.0.4 pip: 21.2.4 conda: None pytest: None IPython: 7.31.1 sphinx: None

 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/6272/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed 13221727 issue

Links from other tables