issue_comments

Table actions

6 rows where author_association = "MEMBER", issue = 868907284 and user = 4160723 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

  • benbovy · 6 ✖

issue 1

  • 'NaT' as fill value and netcdf export · 6 ✖

author_association 1

  • MEMBER · 6 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
831298605 https://github.com/pydata/xarray/issues/5223#issuecomment-831298605 https://api.github.com/repos/pydata/xarray/issues/5223 MDEyOklzc3VlQ29tbWVudDgzMTI5ODYwNQ== benbovy 4160723 2021-05-03T14:31:14Z 2021-05-03T14:32:07Z MEMBER

I guess your last example works because Xarray's Dataset.to_zarr() does some operations (e.g., encoding datetime values as floats + adding encoding attributes like calendar and units) that Xarray-simlab doesn't (Xarray-simlab creates the zarr datasets by directly using the zarr-python API).

The key thing is adding units in variable encoding, e.g., from a Xarray-simlab output dataset:

ds_out.to_netcdf('test.nc', engine='netcdf4', encoding={'p__var': {'units': 'days since 2010-01-01 00:00:00'}})

Maybe we could borrow some logic from Dataset.to_zarr() in Xarray-simlab to encode datetime values when it is saved to zarr.

I think that we can then either close this issue or move it to Xarray-simlab.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   'NaT' as fill value and netcdf export 868907284
831130073 https://github.com/pydata/xarray/issues/5223#issuecomment-831130073 https://api.github.com/repos/pydata/xarray/issues/5223 MDEyOklzc3VlQ29tbWVudDgzMTEzMDA3Mw== benbovy 4160723 2021-05-03T09:09:25Z 2021-05-03T09:09:25Z MEMBER

I checked with a very basic example:

```python import numpy as np import xarray as xr

p_var = np.full((2, 2), np.datetime64('2000-01-01'), dtype='datetime64[ns]') ds = xr.Dataset({'p__var': (('main', 'idx'), var)}) ds.to_netcdf('test.nc', engine='netcdf4') # works! ```

The only difference with the example in your notebook is that in the example above ds.p__var.encoding returns an empty dictionary. If you reset the encoding in your notebook example, e.g., ds_out_no_scale.p__var.encoding = {} you will be able to save the Dataset to a netcdf4 file. Not sure why...

(side note: with mask_and_scale=True, masking missing values withnan causes the dtype to change to float, because type(np.nan) is float).

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   'NaT' as fill value and netcdf export 868907284
829016596 https://github.com/pydata/xarray/issues/5223#issuecomment-829016596 https://api.github.com/repos/pydata/xarray/issues/5223 MDEyOklzc3VlQ29tbWVudDgyOTAxNjU5Ng== benbovy 4160723 2021-04-29T07:50:08Z 2021-04-29T07:50:08Z MEMBER

@jvail could you provide a small reproducible example?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   'NaT' as fill value and netcdf export 868907284
828292425 https://github.com/pydata/xarray/issues/5223#issuecomment-828292425 https://api.github.com/repos/pydata/xarray/issues/5223 MDEyOklzc3VlQ29tbWVudDgyODI5MjQyNQ== benbovy 4160723 2021-04-28T09:11:34Z 2021-04-28T09:11:34Z MEMBER

Opened #5226

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   'NaT' as fill value and netcdf export 868907284
828242346 https://github.com/pydata/xarray/issues/5223#issuecomment-828242346 https://api.github.com/repos/pydata/xarray/issues/5223 MDEyOklzc3VlQ29tbWVudDgyODI0MjM0Ng== benbovy 4160723 2021-04-28T08:02:54Z 2021-04-28T08:03:38Z MEMBER

So maybe the Zarr backend should pop _FillValue from Variable's attrs to encoding even for mask_and_scale=False?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   'NaT' as fill value and netcdf export 868907284
828231169 https://github.com/pydata/xarray/issues/5223#issuecomment-828231169 https://api.github.com/repos/pydata/xarray/issues/5223 MDEyOklzc3VlQ29tbWVudDgyODIzMTE2OQ== benbovy 4160723 2021-04-28T07:47:44Z 2021-04-28T07:47:44Z MEMBER

For more context, xarray-simlab doesn't set the _FillValue attribute directly. Instead it uses Xarray's zarr backend, which leaves the _FillValue item as-is in the variable attributes when it is not picked up in decode_cf (i.e., when setting mask_and_scale=False):

https://github.com/pydata/xarray/blob/0021cdab91f7466f4be0fb32dae92bf3f8290e19/xarray/backends/zarr.py#L369-L372

https://github.com/pydata/xarray/blob/ab4e94ec4f6933476ee0d21c937d8f0f8d39ed82/xarray/coding/variables.py#L183-L186

So maybe the original issue should be solved there?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   'NaT' as fill value and netcdf export 868907284

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);