issue_comments

I think that we can then either close this issue or move it to Xarray-simlab.

Ok. I'll prepare a better example, will try to come up with a suggestion and create an issue over there.

I guess your last example works because Xarray's Dataset.to_zarr() does some operations (e.g., encoding datetime values as floats + adding encoding attributes like calendar and units) that Xarray-simlab doesn't (Xarray-simlab creates the zarr datasets by directly using the zarr-python API).

The key thing is adding units in variable encoding, e.g., from a Xarray-simlab output dataset:

ds_out.to_netcdf('test.nc', engine='netcdf4', encoding={'p__var': {'units': 'days since 2010-01-01 00:00:00'}})

Maybe we could borrow some logic from Dataset.to_zarr() in Xarray-simlab to encode datetime values when it is saved to zarr.

I think that we can then either close this issue or move it to Xarray-simlab.

If you reset the encoding in your notebook example, e.g., ds_out_no_scale.p__var.encoding = {} you will be able to save the Dataset to a netcdf4 file. Not sure why...

Yes, thank you @benbovy , that works for me as well as a work-around.

However .. (excuse my stubbornness):

I tried writing a simple array from ds -> zarr -> nc -> ds. Surprisingly everything works fine. Even the fill value is there (with and w/o mask_and_scale). It might be a xarray simlab issue and something goes wrong when writing/reading to/from zarr. Should I create an issue over there and close this one?

https://github.com/jvail/xarray-simlab/blob/test_encoding_netcdf/notebooks/ds_to_zarr_to_nc_to_ds.ipynb

I checked with a very basic example:

```python import numpy as np import xarray as xr

p_var = np.full((2, 2), np.datetime64('2000-01-01'), dtype='datetime64[ns]') ds = xr.Dataset({'p__var': (('main', 'idx'), var)}) ds.to_netcdf('test.nc', engine='netcdf4') # works! ```

The only difference with the example in your notebook is that in the example above ds.p__var.encoding returns an empty dictionary. If you reset the encoding in your notebook example, e.g., ds_out_no_scale.p__var.encoding = {} you will be able to save the Dataset to a netcdf4 file. Not sure why...

(side note: with mask_and_scale=True, masking missing values withnan causes the dtype to change to float, because type(np.nan) is float).

@jvail could you provide a small reproducible example?

Sure, thank you. I'll put a notebook together

I hope this helps: https://github.com/jvail/xarray-simlab/blob/test_encoding_netcdf/notebooks/test_encoding_netcdf.ipynb

---- edit If I serialize to a dict after popping the _FillValue and then back it seems to work (both the datetime and NaT encoding) unless the datetime64 variable has at least one none 'Nat'.

xr.Dataset.from_dict(ds.to_dict()).to_netcdf('nc', engine='netcdf4')

Removing the entire encoding obj from the variable with dtype datetime64 seems to work as well. However, no idea, if that is a good idea.

@jvail could you provide a small reproducible example?

I guess if I manually drop the attrs._FillValue writing netcdf should work.

Hm, now I get ValueError: unsupported dtype for netCDF4 variable: datetime64[ns] :) I'll see if I can figure out where the problem is. Maybe in zarr since I can write an identical Dataset to netcdf without error.

Opened #5226

Thank you @benbovy! I could not have pinned down the issue myself. I guess if I manually drop the attrs._FillValue writing netcdf should work.

Not sure if this should be closed or maybe moved to xarray-simlab?

Opened #5226

So maybe the Zarr backend should pop _FillValue from Variable's attrs to encoding even for mask_and_scale=False?

For more context, xarray-simlab doesn't set the _FillValue attribute directly. Instead it uses Xarray's zarr backend, which leaves the _FillValue item as-is in the variable attributes when it is not picked up in decode_cf (i.e., when setting mask_and_scale=False):

https://github.com/pydata/xarray/blob/0021cdab91f7466f4be0fb32dae92bf3f8290e19/xarray/backends/zarr.py#L369-L372

https://github.com/pydata/xarray/blob/ab4e94ec4f6933476ee0d21c937d8f0f8d39ed82/xarray/coding/variables.py#L183-L186

So maybe the original issue should be solved there?

OK great! Does that mean this is solved? Or you need it on attrs?

Does putting it on .encoding solve the immediate issue?

The issue is gone if I have neither an attrs._FillValue nor encoding._FillValue or just encoding._FillValue. The issue comes with attrs._FillValue.

Ah so it is a special attrs...

Does putting it on .encoding solve the immediate issue?

Please could I ask once more — forgive me if I'm missing something but I did ask this a week ago and still don't understand #5200 (comment):

Is there a specific reason _FillValue needs to be in the attrs? (I'm not a big netcdf user so there may be)

Sorry, overlooked that one. I'll dig in the simlab code.

Why is xarray not ignoring the _FillValue in attrs?

It's trying to serialize it, as it would any other attrs.

Ok, that would make sense if '_FillValue' would just be an ordinary attr. But it seems it is not. If I set it to a string 'NaT' netcdf complains.

``` lib/python3.8/site-packages/scipy/io/netcdf.py in _get_encoded_fill_value(self) 1030 """ 1031 if '_FillValue' in self._attributes: -> 1032 fill_value = np.array(self._attributes['_FillValue'], 1033 dtype=self.data.dtype).tobytes() 1034 if len(fill_value) == self.itemsize():

ValueError: could not convert string to float: b'NaT' ```

If I set it to 0 I do not get an error. But then - after reloading the netcdf file - my NaT becomes array(['1970-01-01T00:00:00.000000000'], dtype='datetime64[ns]')

Please could I ask once more — forgive me if I'm missing something but I did ask this a week ago and still don't understand https://github.com/pydata/xarray/discussions/5200#discussioncomment-638329:

Is there a specific reason _FillValue needs to be in the attrs? (I'm not a big netcdf user so there may be)

Why is xarray not ignoring the _FillValue in attrs?

It's trying to serialize it, as it would any other attrs.

Re-Hello and thank you,

hm, so the proper way (if necessary at all) would be to set

da.encoding = {'_FillValue': np.datetime64('NAT')}

rather than putting it in attrs?

Maybe I make a mistake in xarray-simlab and I should set the encoding explicitly and my confusion comes from having _FillValue in attrs and encoding. But - as far as I can see - in the xarray docs for writing netcdf the encoding _FillValue is mentioned. Why is xarray not ignoring the _FillValue in attrs?

[xarray-simlab] stores the _FillValue as an attribute which in turn is used by netcdf

that might be a bug in xarray-simlab (cc @benbovy). Usually, the fill value is used to replace missing values on disk. For example, python np.array([0, np.nan, 2, np.nan, np.nan, 5]) with a fill value of -1 could be encoded as [0, -1, 2, -1, -1, 5] before writing to disk, which can be saved as a int (int8, even) instead of a float. Same for datetimes: ["2020-01-01", "NaT", "2020-12-01"] with a fill value of -1 can be encoded as [0, -1, 11] with units = "months since 2020-01-01" and the standard calendar. As far as I understand it, using np.datetime64("NaT") as fill value does not make much sense because netCDF does not support datetime dtypes:

Also, it's strange that _FillValue is saved to attrs and not encoding (which means xarray won't actually use it to encode the arrays).

As a summary, I think you should open this issue on the issue tracker of xarray-simlab.

Also, it's strange that _FillValue is saved to attrs and not encoding (which means xarray won't actually use it to encode the arrays).

I asked for reference for this in #5200. Agree this is surprising.