html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/2037#issuecomment-1063906458,https://api.github.com/repos/pydata/xarray/issues/2037,1063906458,IC_kwDOAMm_X84_aeya,47473455,2022-03-10T10:35:23Z,2022-03-10T10:59:17Z,NONE,"I see know why it did not had sense to push something. Basically cf requirements are in conflict with backend requirements.
It would be needed to restructure tests and review which from backend have sense to apply.
```py
======================================================== short test summary info ========================================================
FAILED xarray/tests/test_backends.py::TestScipyFileObject::test_write_store - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyInMemoryData::test_write_store - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyInMemoryData::test_ondisk_after_print - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyInMemoryData::test_vectorized_indexing - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyInMemoryData::test_isel_dataarray - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyInMemoryData::test_roundtrip_test_data - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyInMemoryData::test_roundtrip_timedelta_data - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFileObject::test_isel_dataarray - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyInMemoryData::test_load - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFilePath::test_write_store - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFileObject::test_coordinate_variables_after_dataset_roundtrip - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyInMemoryData::test_roundtrip_numpy_datetime_data - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFileObject::test_roundtrip_test_data - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyInMemoryData::test_orthogonal_indexing - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFileObject::test_dataset_compute - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFileObject::test_roundtrip_timedelta_data - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyInMemoryData::test_zero_dimensional_variable - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyInMemoryData::test_dataset_compute - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyInMemoryData::test_array_type_after_indexing - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyInMemoryData::test_append_write - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFilePath::test_vectorized_indexing - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyInMemoryData::test_coordinate_variables_after_dataset_roundtrip - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFilePath::test_isel_dataarray - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFileObject::test_append_overwrite_values - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFileObject::test_load - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFileObject::test_grid_mapping_and_bounds_are_coordinates_after_dataarray_roundtrip - Ke...
FAILED xarray/tests/test_backends.py::TestGenericNetCDFData::test_cross_engine_read_write_netcdf3 - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFileObject::test_vectorized_indexing - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyInMemoryData::test_grid_mapping_and_bounds_are_coordinates_after_dataarray_roundtrip - ...
FAILED xarray/tests/test_backends.py::TestScipyInMemoryData::test_append_overwrite_values - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFileObject::test_zero_dimensional_variable - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFileObject::test_ondisk_after_print - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFilePath::test_roundtrip_timedelta_data - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFileObject::test_append_with_invalid_dim_raises - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFileObject::test_orthogonal_indexing - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFileObject::test_array_type_after_indexing - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFilePath::test_orthogonal_indexing - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyInMemoryData::test_append_with_invalid_dim_raises - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFileObject::test_roundtrip_numpy_datetime_data - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFilePath::test_zero_dimensional_variable - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFilePath::test_roundtrip_test_data - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFileObject::test_append_write - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFilePath::test_ondisk_after_print - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFilePath::test_roundtrip_numpy_datetime_data - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFilePath::test_dataset_compute - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFilePath::test_append_write - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFilePath::test_load - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFilePath::test_array_type_after_indexing - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestGenericNetCDFData::test_engine - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFilePath::test_grid_mapping_and_bounds_are_coordinates_after_dataarray_roundtrip - KeyE...
FAILED xarray/tests/test_backends.py::TestScipyFilePath::test_coordinate_variables_after_dataset_roundtrip - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFilePath::test_append_with_invalid_dim_raises - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::TestScipyFilePath::test_append_overwrite_values - KeyError: ('O', 8)
FAILED xarray/tests/test_backends.py::test_scipy_entrypoint - KeyError: ('O', 8)
```
However your snipet worked, but it is not applying to boundary variables, which is a cf requirement as well.
Probably I would edit it to something like:
```py
# remove _FillValue for coordinate variables as missing values are not permitted (issue #2037)
# a coordinate variable is a one-dimensional variable with the same name as its dimension
# see coordinate variable in http://cfconventions.org/cf-conventions/cf-conventions.html#terminology
for var in new_vars.keys():
if new_vars[var].dims == (var,):
new_vars[var].attrs[""_FillValue""] = None
if ""bounds"" in new_vars[var].attrs:
bnds = new_vars[var].attrs['bounds']
new_vars[bnds].attrs[""_FillValue""] = None
```
I will create a PR with the hope a developer will take a look into this.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,310833761
https://github.com/pydata/xarray/issues/2037#issuecomment-1063808399,https://api.github.com/repos/pydata/xarray/issues/2037,1063808399,IC_kwDOAMm_X84_aG2P,40183561,2022-03-10T08:46:31Z,2022-03-10T08:46:31Z,CONTRIBUTOR,"@BorjaEst I don't think I ever pushed anything up, sorry. Probably best to start a new one.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,310833761
https://github.com/pydata/xarray/issues/2037#issuecomment-1063806501,https://api.github.com/repos/pydata/xarray/issues/2037,1063806501,IC_kwDOAMm_X84_aGYl,47473455,2022-03-10T08:44:15Z,2022-03-10T08:44:15Z,NONE,"> @BorjaEst I haven't opened a PR for this slightly_smiling_face
@ellesmith88 is there a branch/fork where you implemented your trials or shall I start one and add your code?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,310833761
https://github.com/pydata/xarray/issues/2037#issuecomment-1063784085,https://api.github.com/repos/pydata/xarray/issues/2037,1063784085,IC_kwDOAMm_X84_aA6V,40183561,2022-03-10T08:14:59Z,2022-03-10T08:14:59Z,CONTRIBUTOR,@BorjaEst I haven't opened a PR for this 🙂 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,310833761
https://github.com/pydata/xarray/issues/2037#issuecomment-1063707401,https://api.github.com/repos/pydata/xarray/issues/2037,1063707401,IC_kwDOAMm_X84_ZuMJ,47473455,2022-03-10T06:24:58Z,2022-03-10T06:25:12Z,NONE,"@ellesmith88 [ellesmith88](https://github.com/ellesmith88)
> but I couldn't quite get this to work, with some tests in test_backends.py failing. I could see that the fill value is being added in by maybe_default_fill_value in conventions.py so maybe it would be better to skip coordinate variable and bounds here. I can't really spend anymore time on this but putting it here in case its useful.
I see you are setting `_FillValue` to `None` but probably it should be set to `False` instead.
See https://stackoverflow.com/questions/45693688/xarray-automatically-applying-fillvalue-to-coordinates-on-netcdf-output
I can give it a try. In which PR are you working on this?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,310833761
https://github.com/pydata/xarray/issues/2037#issuecomment-1063705114,https://api.github.com/repos/pydata/xarray/issues/2037,1063705114,IC_kwDOAMm_X84_Ztoa,47473455,2022-03-10T06:20:40Z,2022-03-10T06:20:40Z,NONE,"I am posting here a temporal solution to this in case someone has the same issue.
Based on https://stackoverflow.com/questions/45693688/xarray-automatically-applying-fillvalue-to-coordinates-on-netcdf-output
```py
for var in ds.variables:
if '_FillValue' not in ds[var].attrs:
ds[var].attrs['_FillValue'] = False
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,310833761
https://github.com/pydata/xarray/issues/2037#issuecomment-857563427,https://api.github.com/repos/pydata/xarray/issues/2037,857563427,MDEyOklzc3VlQ29tbWVudDg1NzU2MzQyNw==,40183561,2021-06-09T10:03:22Z,2021-06-10T12:39:04Z,CONTRIBUTOR,"I had a go at working on this, as mentioned in #5448 it could be a fix in https://github.com/pydata/xarray/blob/ce01f42134962b63d453657c5cb649ebf152283d/xarray/conventions.py#L707
I couldn't see how to fix it in there but tried a fix at the end of https://github.com/pydata/xarray/blob/ce01f42134962b63d453657c5cb649ebf152283d/xarray/conventions.py#L813
for bounds and coordinate variables. See below:
```python
def cf_encoder(variables, attributes):
""""""
Encode a set of CF encoded variables and attributes.
Takes a dicts of variables and attributes and encodes them
to conform to CF conventions as much as possible.
This includes masking, scaling, character array handling,
and CF-time encoding.
Parameters
----------
variables : dict
A dictionary mapping from variable name to xarray.Variable
attributes : dict
A dictionary mapping from attribute name to value
Returns
-------
encoded_variables : dict
A dictionary mapping from variable name to xarray.Variable,
encoded_attributes : dict
A dictionary mapping from attribute name to value
See Also
--------
decode_cf_variable, encode_cf_variable
""""""
# add encoding for time bounds variables if present.
_update_bounds_encoding(variables)
new_vars = {k: encode_cf_variable(v, name=k) for k, v in variables.items()}
# Remove attrs from bounds variables (issue #2921)
for var in new_vars.values():
bounds = var.attrs[""bounds""] if ""bounds"" in var.attrs else None
if bounds and bounds in new_vars:
# see http://cfconventions.org/cf-conventions/cf-conventions.html#cell-boundaries
for attr in [
""units"",
""standard_name"",
""axis"",
""positive"",
""calendar"",
""long_name"",
""leap_month"",
""leap_year"",
""month_lengths"",
]:
if attr in new_vars[bounds].attrs and attr in var.attrs:
if new_vars[bounds].attrs[attr] == var.attrs[attr]:
new_vars[bounds].attrs.pop(attr)
# remove _FillValue for bounds
if new_vars[bounds].attrs.get(""_FillValue""):
new_vars[bounds].attrs[""_FillValue""] = None
# remove _FillValue for coordinate variables as missing values are not permitted
# a coordinate variable is a one-dimensional variable with the same name as its dimension
# see coordinate variable in http://cfconventions.org/cf-conventions/cf-conventions.html#terminology
for var in new_vars.keys():
if new_vars[var].dims == (var,) and new_vars[var].attrs.get(""_FillValue""):
new_vars[var].attrs[""_FillValue""] = None
return new_vars, attributes
```
tests to add into `test_conventions.py` could be something like:
```python
def test_fill_value_none_for_coordinate_variables():
# need to be floats so _FillValue is added
temp = 273.15 + 25 * np.random.randn(2, 2)
lon = [0.0, 5.0]
lat = [10.0, 20.0]
ds = Dataset({'temperature': (['lat', 'lon'], temp)},
coords={'lat': lat,
'lon': lon})
ds['lat'].attrs = {
'standard_name': 'latitude',
'long_name': 'latitude',
'units': 'degrees_north',
'axis': 'Y'}
ds['lon'].attrs = {
'standard_name': 'longitude',
'long_name': 'longitude',
'units': 'degrees_east',
'axis': 'X'}
ds['temperature'].attrs = {
'standard_name': 'air_temperature',
'units': 'K'}
v, _ = conventions.cf_encoder(ds.variables, ds.attrs)
assert v[""lat""].attrs['_FillValue'] is None
assert v[""lon""].attrs['_FillValue'] is None
# check _FillValue still inserted for temperature
assert v[""temperature""].attrs['_FillValue'] is not None
def test_fill_value_none_for_bounds_variables():
# need to be floats so _FillValue is added
temp = 273.15 + 25 * np.random.randn(2, 2)
lat_bnds = np.arange(4.0).reshape((2, 2))
lon = [0.0, 5.0]
lat = [10.0, 20.0]
ds = Dataset({'temperature': (['lat', 'lon'], temp),
'lat_bnds': (['lat', 'nbnds'], lat_bnds),
'lon_bnds': (['lon', 'nbnds'], lat_bnds)},
coords={'lat': lat,
'lon': lon})
ds['lat'].attrs = {
'standard_name': 'latitude',
'long_name': 'latitude',
'units': 'degrees_north',
'axis': 'Y',
'bounds': 'lat_bnds'}
ds['lon'].attrs = {
'standard_name': 'longitude',
'long_name': 'longitude',
'units': 'degrees_east',
'axis': 'X',
'bounds': 'lon_bnds'}
ds['temperature'].attrs = {
'standard_name': 'air_temperature',
'units': 'K'}
v, _ = conventions.cf_encoder(ds.variables, ds.attrs)
assert v[""lat_bnds""].attrs['_FillValue'] is None
assert v[""lon_bnds""].attrs['_FillValue'] is None
```
but I couldn't quite get this to work, with some tests in `test_backends.py` failing. I could see that the fill value is being added in by `maybe_default_fill_value` in `conventions.py` so maybe it would be better to skip coordinate variable and bounds here. I can't really spend anymore time on this but putting it here in case its useful.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,310833761
https://github.com/pydata/xarray/issues/2037#issuecomment-856085619,https://api.github.com/repos/pydata/xarray/issues/2037,856085619,MDEyOklzc3VlQ29tbWVudDg1NjA4NTYxOQ==,2448579,2021-06-07T16:31:00Z,2021-06-07T16:31:00Z,MEMBER,From #5448 we should skip it for bounds variables too.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,310833761