issues

Table actions

1 row where state = "closed", type = "issue" and user = 40183561 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date), closed_at (date)

type 1

  • issue · 1 ✖

state 1

  • closed · 1 ✖

repo 1

id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
913431220 MDU6SXNzdWU5MTM0MzEyMjA= 5448 _FillValue added with output to netcdf ellesmith88 40183561 closed 0     1 2021-06-07T11:10:57Z 2021-06-07T16:43:04Z 2021-06-07T16:43:04Z CONTRIBUTOR      

What happened: When opening a netCDF file with xarray, then outputting to netCDF, _FillValue is added. This is an issue for coordinate variables time, lat and lon as CF conventions say that coordinate variables cannot have missing values, so we don't want this added _FillValue in this case.

e.g.

Before working with xarray: double lat(lat) ; lat:bounds = "lat_bnds" ; lat:units = "degrees_north" ; lat:axis = "Y" ; lat:long_name = "latitude" ; lat:standard_name = "latitude" ;

After working with xarray: double lat(lat) ; lat:_FillValue = NaN ; lat:bounds = "lat_bnds" ; lat:units = "degrees_north" ; lat:axis = "Y" ; lat:long_name = "latitude" ; lat:standard_name = "latitude" ;

_FIllValue is also added to the netCDF file for other variables but the main issue is the coordinate variables.

By setting ds.lat.encoding['_FillValue'] = None before outputting to NetCDF, the lat:_FillValue = NaN ; can be removed, but I don't think it should be there in the first place?

What you expected to happen: Output in file after to_netcdf to be the same as the input file.

Minimal Complete Verifiable Example:

```python import xarray as xr

my_file = xr.open_dataset("ts_Amon_HadGEM3-GC31-LL_ssp126_r1i1p1f3_gn_201501-204912.nc", use_cftime=True) my_file.to_netcdf("my_file_output.nc") ``` The file I used can be retrieved using https://dap.ceda.ac.uk/badc/cmip6/data/CMIP6/ScenarioMIP/MOHC/HadGEM3-GC31-LL/ssp126/r1i1p1f3/Amon/ts/gn/v20200114/ts_Amon_HadGEM3-GC31-LL_ssp126_r1i1p1f3_gn_201501-204912.nc, but this happens with any file.

Then comparing the ncdumps:

ncdump -h ts_Amon_HadGEM3-GC31-LL_ssp126_r1i1p1f3_gn_201501-204912.nc netcdf ts_Amon_HadGEM3-GC31-LL_ssp126_r1i1p1f3_gn_201501-204912 { dimensions: time = UNLIMITED ; // (420 currently) bnds = 2 ; lat = 144 ; lon = 192 ; variables: double time(time) ; time:bounds = "time_bnds" ; time:units = "days since 1850-01-01" ; time:calendar = "360_day" ; time:axis = "T" ; time:long_name = "time" ; time:standard_name = "time" ; double time_bnds(time, bnds) ; double lat(lat) ; lat:bounds = "lat_bnds" ; lat:units = "degrees_north" ; lat:axis = "Y" ; lat:long_name = "Latitude" ; lat:standard_name = "latitude" ; double lat_bnds(lat, bnds) ; double lon(lon) ; lon:bounds = "lon_bnds" ; lon:units = "degrees_east" ; lon:axis = "X" ; lon:long_name = "Longitude" ; lon:standard_name = "longitude" ; double lon_bnds(lon, bnds) ; float ts(time, lat, lon) ; ts:standard_name = "surface_temperature" ; ts:long_name = "Surface Temperature" ; ts:comment = "Temperature of the lower boundary of the atmosphere" ; ts:units = "K" ; ts:original_name = "mo: (stash: m01s00i024, lbproc: 128)" ; ts:cell_methods = "area: time: mean" ; ts:cell_measures = "area: areacella" ; ts:history = "2020-01-13T10:05:19Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20)." ; ts:missing_value = 1.e+20f ; ts:_FillValue = 1.e+20f ;
and

ncdump -h my_file_output.nc netcdf my_file_output { dimensions: time = UNLIMITED ; // (420 currently) bnds = 2 ; lat = 144 ; lon = 192 ; variables: double time(time) ; time:_FillValue = NaN ; time:bounds = "time_bnds" ; time:axis = "T" ; time:long_name = "time" ; time:standard_name = "time" ; time:units = "days since 1850-01-01" ; time:calendar = "360_day" ; double time_bnds(time, bnds) ; time_bnds:_FillValue = NaN ; double lat(lat) ; lat:_FillValue = NaN ; lat:bounds = "lat_bnds" ; lat:units = "degrees_north" ; lat:axis = "Y" ; lat:long_name = "Latitude" ; lat:standard_name = "latitude" ; double lat_bnds(lat, bnds) ; lat_bnds:_FillValue = NaN ; double lon(lon) ; lon:_FillValue = NaN ; lon:bounds = "lon_bnds" ; lon:units = "degrees_east" ; lon:axis = "X" ; lon:long_name = "Longitude" ; lon:standard_name = "longitude" ; double lon_bnds(lon, bnds) ; lon_bnds:_FillValue = NaN ; float ts(time, lat, lon) ; ts:_FillValue = 1.e+20f ; ts:standard_name = "surface_temperature" ; ts:long_name = "Surface Temperature" ; ts:comment = "Temperature of the lower boundary of the atmosphere" ; ts:units = "K" ; ts:original_name = "mo: (stash: m01s00i024, lbproc: 128)" ; ts:cell_methods = "area: time: mean" ; ts:cell_measures = "area: areacella" ; ts:history = "2020-01-13T10:05:19Z altered by CMOR: replaced missing value flag (-1.07374e+09) with standard missing value (1e+20)." ; ts:missing_value = 1.e+20f ;

shows that _FillValue has been added to time, time_bnds, lat, lat_bnds, lon and lon_bnds

Environment:

Output of <tt>xr.show_versions()</tt> INSTALLED VERSIONS ------------------ commit: None python: 3.7.3 (default, Mar 27 2019, 16:54:48) [Clang 4.0.1 (tags/RELEASE_401/final)] python-bits: 64 OS: Darwin OS-release: 18.7.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.10.5 libnetcdf: 4.6.3 xarray: 0.18.2 pandas: 1.1.3 numpy: 1.19.2 scipy: None netCDF4: 1.5.4 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.4.1 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: 1.3.2 dask: 2.30.0 distributed: 2.30.0 matplotlib: None cartopy: None seaborn: None numbagg: None pint: None setuptools: 54.1.1 pip: 21.0.1 conda: None pytest: 6.2.2 IPython: 7.21.0 sphinx: 1.8.1 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/5448/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed xarray 13221727 issue

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [active_lock_reason] TEXT,
   [draft] INTEGER,
   [pull_request] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [state_reason] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
);
CREATE INDEX [idx_issues_repo]
    ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
    ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
    ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
    ON [issues] ([user]);