html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/5223#issuecomment-831298605,https://api.github.com/repos/pydata/xarray/issues/5223,831298605,MDEyOklzc3VlQ29tbWVudDgzMTI5ODYwNQ==,4160723,2021-05-03T14:31:14Z,2021-05-03T14:32:07Z,MEMBER,"I guess your last example works because Xarray's `Dataset.to_zarr()` does some operations (e.g., encoding datetime values as floats + adding encoding attributes like `calendar` and `units`) that Xarray-simlab doesn't (Xarray-simlab creates the zarr datasets by directly using the zarr-python API).
The key thing is adding `units` in variable encoding, e.g., from a Xarray-simlab output dataset:
```
ds_out.to_netcdf('test.nc', engine='netcdf4', encoding={'p__var': {'units': 'days since 2010-01-01 00:00:00'}})
```
Maybe we could borrow some logic from `Dataset.to_zarr()` in Xarray-simlab to encode datetime values when it is saved to zarr.
I think that we can then either close this issue or move it to Xarray-simlab.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,868907284
https://github.com/pydata/xarray/issues/5223#issuecomment-831130073,https://api.github.com/repos/pydata/xarray/issues/5223,831130073,MDEyOklzc3VlQ29tbWVudDgzMTEzMDA3Mw==,4160723,2021-05-03T09:09:25Z,2021-05-03T09:09:25Z,MEMBER,"I checked with a very basic example:
```python
import numpy as np
import xarray as xr
p_var = np.full((2, 2), np.datetime64('2000-01-01'), dtype='datetime64[ns]')
ds = xr.Dataset({'p__var': (('main', 'idx'), var)})
ds.to_netcdf('test.nc', engine='netcdf4') # works!
```
The only difference with the example in your notebook is that in the example above `ds.p__var.encoding` returns an empty dictionary. If you reset the encoding in your notebook example, e.g., `ds_out_no_scale.p__var.encoding = {}` you will be able to save the Dataset to a netcdf4 file. Not sure why...
(side note: with `mask_and_scale=True`, masking missing values with`nan` causes the dtype to change to `float`, because `type(np.nan)` is `float`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,868907284
https://github.com/pydata/xarray/issues/5223#issuecomment-829016596,https://api.github.com/repos/pydata/xarray/issues/5223,829016596,MDEyOklzc3VlQ29tbWVudDgyOTAxNjU5Ng==,4160723,2021-04-29T07:50:08Z,2021-04-29T07:50:08Z,MEMBER,"@jvail could you provide a small reproducible example?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,868907284
https://github.com/pydata/xarray/issues/5223#issuecomment-828292425,https://api.github.com/repos/pydata/xarray/issues/5223,828292425,MDEyOklzc3VlQ29tbWVudDgyODI5MjQyNQ==,4160723,2021-04-28T09:11:34Z,2021-04-28T09:11:34Z,MEMBER,Opened #5226,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,868907284
https://github.com/pydata/xarray/issues/5223#issuecomment-828242346,https://api.github.com/repos/pydata/xarray/issues/5223,828242346,MDEyOklzc3VlQ29tbWVudDgyODI0MjM0Ng==,4160723,2021-04-28T08:02:54Z,2021-04-28T08:03:38Z,MEMBER,So maybe the Zarr backend should pop `_FillValue` from Variable's `attrs` to `encoding` even for `mask_and_scale=False`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,868907284
https://github.com/pydata/xarray/issues/5223#issuecomment-828231169,https://api.github.com/repos/pydata/xarray/issues/5223,828231169,MDEyOklzc3VlQ29tbWVudDgyODIzMTE2OQ==,4160723,2021-04-28T07:47:44Z,2021-04-28T07:47:44Z,MEMBER,"For more context, xarray-simlab doesn't set the `_FillValue` attribute directly. Instead it uses Xarray's zarr backend, which leaves the `_FillValue` item as-is in the variable attributes when it is not picked up in `decode_cf` (i.e., when setting `mask_and_scale=False`):
https://github.com/pydata/xarray/blob/0021cdab91f7466f4be0fb32dae92bf3f8290e19/xarray/backends/zarr.py#L369-L372
https://github.com/pydata/xarray/blob/ab4e94ec4f6933476ee0d21c937d8f0f8d39ed82/xarray/coding/variables.py#L183-L186
So maybe the original issue should be solved there?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,868907284
https://github.com/pydata/xarray/issues/5223#issuecomment-828230957,https://api.github.com/repos/pydata/xarray/issues/5223,828230957,MDEyOklzc3VlQ29tbWVudDgyODIzMDk1Nw==,5635139,2021-04-28T07:47:25Z,2021-04-28T07:47:25Z,MEMBER,OK great! Does that mean this is solved? Or you need it on `attrs`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,868907284
https://github.com/pydata/xarray/issues/5223#issuecomment-828162977,https://api.github.com/repos/pydata/xarray/issues/5223,828162977,MDEyOklzc3VlQ29tbWVudDgyODE2Mjk3Nw==,5635139,2021-04-28T05:48:25Z,2021-04-28T05:48:25Z,MEMBER,"Ah so it is a special attrs...
Does putting it on `.encoding` solve the immediate issue?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,868907284
https://github.com/pydata/xarray/issues/5223#issuecomment-828156283,https://api.github.com/repos/pydata/xarray/issues/5223,828156283,MDEyOklzc3VlQ29tbWVudDgyODE1NjI4Mw==,5635139,2021-04-28T05:32:20Z,2021-04-28T05:32:20Z,MEMBER,"Please could I ask once more — forgive me if I'm missing something but I did ask this a week ago and still don't understand https://github.com/pydata/xarray/discussions/5200#discussioncomment-638329:
> Is there a specific reason _FillValue needs to be in the attrs? (I'm not a big netcdf user so there may be)
---
> Why is xarray not ignoring the _FillValue in attrs?
It's trying to serialize it, as it would any other attrs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,868907284
https://github.com/pydata/xarray/issues/5223#issuecomment-827675422,https://api.github.com/repos/pydata/xarray/issues/5223,827675422,MDEyOklzc3VlQ29tbWVudDgyNzY3NTQyMg==,14808389,2021-04-27T15:00:40Z,2021-04-27T18:15:02Z,MEMBER,"> [`xarray-simlab`] stores the `_FillValue` as an attribute which in turn is used by netcdf
that might be a bug in `xarray-simlab` (cc @benbovy). Usually, the fill value is used to replace missing values on disk. For example,
```python
np.array([0, np.nan, 2, np.nan, np.nan, 5])
```
with a fill value of `-1` could be encoded as `[0, -1, 2, -1, -1, 5]` before writing to disk, which can be saved as a `int` (`int8`, even) instead of a `float`. Same for datetimes: `[""2020-01-01"", ""NaT"", ""2020-12-01""]` with a fill value of `-1` can be encoded as `[0, -1, 11]` with `units = ""months since 2020-01-01""` and the standard calendar. As far as I understand it, using `np.datetime64(""NaT"")` as fill value does not make much sense because netCDF does not support datetime dtypes:
traceback when trying to save a datetime array attribute
```pytb
TypeError Traceback (most recent call last)
in
1 import numpy as np
2 import xarray as xr
----> 3 xr.Dataset(attrs={""_FillValue"": np.array(""NaT"", dtype=""M"")}).to_netcdf(""test.nc"")
.../xarray/core/dataset.py in to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute, invalid_netcdf)
1752 from ..backends.api import to_netcdf
1753
-> 1754 return to_netcdf(
1755 self,
1756 path,
.../xarray/backends/api.py in to_netcdf(dataset, path_or_file, mode, format, group, engine, encoding, unlimited_dims, compute, multifile, invalid_netcdf)
1066 # TODO: allow this work (setting up the file for writing array data)
1067 # to be parallelized with dask
-> 1068 dump_to_store(
1069 dataset, store, writer, encoding=encoding, unlimited_dims=unlimited_dims
1070 )
.../xarray/backends/api.py in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims)
1113 variables, attrs = encoder(variables, attrs)
1114
-> 1115 store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
1116
1117
.../xarray/backends/common.py in store(self, variables, attributes, check_encoding_set, writer, unlimited_dims)
263 variables, attributes = self.encode(variables, attributes)
264
--> 265 self.set_attributes(attributes)
266 self.set_dimensions(variables, unlimited_dims=unlimited_dims)
267 self.set_variables(
.../xarray/backends/common.py in set_attributes(self, attributes)
280 """"""
281 for k, v in attributes.items():
--> 282 self.set_attribute(k, v)
283
284 def set_variables(self, variables, check_encoding_set, writer, unlimited_dims=None):
.../xarray/backends/netCDF4_.py in set_attribute(self, key, value)
449 self.ds.setncattr_string(key, value)
450 else:
--> 451 self.ds.setncattr(key, value)
452
453 def encode_variable(self, variable):
src/netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dataset.setncattr()
src/netCDF4/_netCDF4.pyx in netCDF4._netCDF4._set_att()
TypeError: illegal data type for attribute b'_FillValue', must be one of dict_keys(['S1', 'i1', 'u1', 'i2', 'u2', 'i4', 'u4', 'i8', 'u8', 'f4', 'f8']), got M8
```