issue_comments

I guess your last example works because Xarray's Dataset.to_zarr() does some operations (e.g., encoding datetime values as floats + adding encoding attributes like calendar and units) that Xarray-simlab doesn't (Xarray-simlab creates the zarr datasets by directly using the zarr-python API).

The key thing is adding units in variable encoding, e.g., from a Xarray-simlab output dataset:

ds_out.to_netcdf('test.nc', engine='netcdf4', encoding={'p__var': {'units': 'days since 2010-01-01 00:00:00'}})

Maybe we could borrow some logic from Dataset.to_zarr() in Xarray-simlab to encode datetime values when it is saved to zarr.

I think that we can then either close this issue or move it to Xarray-simlab.

I checked with a very basic example:

```python import numpy as np import xarray as xr

p_var = np.full((2, 2), np.datetime64('2000-01-01'), dtype='datetime64[ns]') ds = xr.Dataset({'p__var': (('main', 'idx'), var)}) ds.to_netcdf('test.nc', engine='netcdf4') # works! ```

The only difference with the example in your notebook is that in the example above ds.p__var.encoding returns an empty dictionary. If you reset the encoding in your notebook example, e.g., ds_out_no_scale.p__var.encoding = {} you will be able to save the Dataset to a netcdf4 file. Not sure why...

(side note: with mask_and_scale=True, masking missing values withnan causes the dtype to change to float, because type(np.nan) is float).

@jvail could you provide a small reproducible example?

Opened #5226

So maybe the Zarr backend should pop _FillValue from Variable's attrs to encoding even for mask_and_scale=False?

For more context, xarray-simlab doesn't set the _FillValue attribute directly. Instead it uses Xarray's zarr backend, which leaves the _FillValue item as-is in the variable attributes when it is not picked up in decode_cf (i.e., when setting mask_and_scale=False):

https://github.com/pydata/xarray/blob/0021cdab91f7466f4be0fb32dae92bf3f8290e19/xarray/backends/zarr.py#L369-L372

https://github.com/pydata/xarray/blob/ab4e94ec4f6933476ee0d21c937d8f0f8d39ed82/xarray/coding/variables.py#L183-L186

So maybe the original issue should be solved there?

OK great! Does that mean this is solved? Or you need it on attrs?

Ah so it is a special attrs...

Does putting it on .encoding solve the immediate issue?

Please could I ask once more — forgive me if I'm missing something but I did ask this a week ago and still don't understand https://github.com/pydata/xarray/discussions/5200#discussioncomment-638329:

Is there a specific reason _FillValue needs to be in the attrs? (I'm not a big netcdf user so there may be)

Why is xarray not ignoring the _FillValue in attrs?

It's trying to serialize it, as it would any other attrs.

[xarray-simlab] stores the _FillValue as an attribute which in turn is used by netcdf

that might be a bug in xarray-simlab (cc @benbovy). Usually, the fill value is used to replace missing values on disk. For example, python np.array([0, np.nan, 2, np.nan, np.nan, 5]) with a fill value of -1 could be encoded as [0, -1, 2, -1, -1, 5] before writing to disk, which can be saved as a int (int8, even) instead of a float. Same for datetimes: ["2020-01-01", "NaT", "2020-12-01"] with a fill value of -1 can be encoded as [0, -1, 11] with units = "months since 2020-01-01" and the standard calendar. As far as I understand it, using np.datetime64("NaT") as fill value does not make much sense because netCDF does not support datetime dtypes:

Also, it's strange that _FillValue is saved to attrs and not encoding (which means xarray won't actually use it to encode the arrays).

As a summary, I think you should open this issue on the issue tracker of xarray-simlab.

Also, it's strange that _FillValue is saved to attrs and not encoding (which means xarray won't actually use it to encode the arrays).

I asked for reference for this in #5200. Agree this is surprising.