html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/1576#issuecomment-330464740,https://api.github.com/repos/pydata/xarray/issues/1576,330464740,MDEyOklzc3VlQ29tbWVudDMzMDQ2NDc0MA==,206773,2017-09-19T08:16:43Z,2017-09-19T08:16:43Z,NONE,"@shoyer
> We currently decode anything with a `_FillValue` attribute to float, ...
I believe this fact is surprising for any user of integer/index/enum/classification datasets. Since its justification seems to be an implementation detail which comes at the cost of increased memory and CPU consumption I suggest documenting it in `open_dataset()` and `decode_cf()` functions.
Here is how we overcome this issue by deleting the `_FillValue` attribute of integer variables if their `scale_factor` and `add_offset` attributes are not provided:
ds = xr.open_dataset(path, decode_cf=False)
old_fill_values = unset_fill_value_for_int_vars(ds)
ds = xr.decode_cf(ds)
reset_fill_value_for_int_vars(ds, old_fill_values)
where `old_fill_values` is a mapping of variable names to fill values.
","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,258500654
https://github.com/pydata/xarray/issues/1576#issuecomment-330275698,https://api.github.com/repos/pydata/xarray/issues/1576,330275698,MDEyOklzc3VlQ29tbWVudDMzMDI3NTY5OA==,206773,2017-09-18T16:20:33Z,2017-09-18T16:20:33Z,NONE,"@jhamman `_NoFill` is about optimizing writes, see [nc_set_fill](https://www.unidata.ucar.edu/software/netcdf/netcdf-4/newdocs/netcdf-c/nc_005fset_005ffill.html)","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,258500654
https://github.com/pydata/xarray/issues/1576#issuecomment-330273842,https://api.github.com/repos/pydata/xarray/issues/1576,330273842,MDEyOklzc3VlQ29tbWVudDMzMDI3Mzg0Mg==,206773,2017-09-18T16:13:45Z,2017-09-18T16:13:45Z,NONE,"I see, that is what is done in `mask_and_scale()`. Why can't xarray used masked arrays, that would retain the original dtype? (Dask, I guess?)
Expanding integers to 8 byte floats not only cost memory but also CPU, including an inaccurate in-memory integer representation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,258500654
https://github.com/pydata/xarray/issues/1576#issuecomment-330267397,https://api.github.com/repos/pydata/xarray/issues/1576,330267397,MDEyOklzc3VlQ29tbWVudDMzMDI2NzM5Nw==,206773,2017-09-18T15:52:55Z,2017-09-18T16:00:01Z,NONE,"I guess, the poblem is caused in [xarray/conventions.py](https://github.com/pydata/xarray/blob/master/xarray/conventions.py#L881).
Note, when debugging into it, `fill_value == nd.array([0], dtype == np.int8)` and `fill_value.dtype.kind='i'` and the latter kind is not dealt with. Therefore `int8` is turned into `float64`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,258500654
https://github.com/pydata/xarray/issues/1576#issuecomment-330261323,https://api.github.com/repos/pydata/xarray/issues/1576,330261323,MDEyOklzc3VlQ29tbWVudDMzMDI2MTMyMw==,206773,2017-09-18T15:32:27Z,2017-09-18T15:32:27Z,NONE,"Here you are
$ ncdump -h -s
netcdf ESACCI-LC-L4-LCCS-Map-300m-P5Y-2005-v1.6.1 {
dimensions:
lat = 64800 ;
lon = 129600 ;
variables:
byte lccs_class(lat, lon) ;
lccs_class:long_name = ""Land cover class defined in LCCS"" ;
lccs_class:standard_name = ""land_cover_lccs"" ;
lccs_class:flag_values = 0b, 10b, 11b, 12b, 20b, 30b, 40b, 50b, 60b, 61b, 62b, 70b, 71b, 72b, 80b, 81b, 82b, 90b, 100b, 110b, 120b, 121b, 122b, -126b, -116b, -106b, -104b, -103b, -96b, -86b, -76b, -66b, -56b, -55b, -54b, -46b,
-36b ;
lccs_class:flag_meanings = ""no_data cropland_rainfed cropland_rainfed_herbaceous_cover cropland_rainfed_tree_or_shrub_cover cropland_irrigated mosaic_cropland mosaic_natural_vegetation tree_broadleaved_evergreen_closed_to_open
tree_broadleaved_deciduous_closed_to_open tree_broadleaved_deciduous_closed tree_broadleaved_deciduous_open tree_needleleaved_evergreen_closed_to_open tree_needleleaved_evergreen_closed tree_needleleaved_evergreen_open tree_needleleaved_decidu
ous_closed_to_open tree_needleleaved_deciduous_closed tree_needleleaved_deciduous_open tree_mixed mosaic_tree_and_shrub mosaic_herbaceous shrubland shrubland_evergreen shrubland_deciduous grassland lichens_and_mosses sparse_vegetation sparse_s
hrub sparse_herbaceous tree_cover_flooded_fresh_or_brakish_water tree_cover_flooded_saline_water shrub_or_herbaceous_cover_flooded urban bare_areas bare_areas_consolidated bare_areas_unconsolidated water snow_and_ice"" ;
lccs_class:valid_min = 1 ;
lccs_class:valid_max = 220 ;
lccs_class:_Unsigned = ""true"" ;
lccs_class:_FillValue = 0b ;
lccs_class:ancillary_variables = ""processed_flag current_pixel_state observation_count algorithmic_confidence_level"" ;
lccs_class:_Storage = ""chunked"" ;
lccs_class:_ChunkSizes = 2048, 2048 ;
lccs_class:_DeflateLevel = 6 ;
lccs_class:_NoFill = ""true"" ;
byte processed_flag(lat, lon) ;
processed_flag:standard_name = ""land_cover_lccs status_flag"" ;
processed_flag:flag_values = 0b, 1b ;
processed_flag:flag_meanings = ""not_processed processed"" ;
processed_flag:valid_min = 0 ;
processed_flag:valid_max = 1 ;
processed_flag:_FillValue = -1b ;
processed_flag:long_name = ""LC map processed area flag"" ;
processed_flag:_Storage = ""chunked"" ;
processed_flag:_ChunkSizes = 2048, 2048 ;
processed_flag:_DeflateLevel = 6 ;
processed_flag:_NoFill = ""true"" ;
byte current_pixel_state(lat, lon) ;
current_pixel_state:standard_name = ""land_cover_lccs status_flag"" ;
current_pixel_state:flag_values = 0b, 1b, 2b, 3b, 4b, 5b ;
current_pixel_state:flag_meanings = ""invalid clear_land clear_water clear_snow_ice cloud cloud_shadow"" ;
current_pixel_state:valid_min = 0 ;
current_pixel_state:valid_max = 5 ;
current_pixel_state:_FillValue = -1b ;
current_pixel_state:long_name = ""LC pixel type mask"" ;
current_pixel_state:_Storage = ""chunked"" ;
current_pixel_state:_ChunkSizes = 2048, 2048 ;
current_pixel_state:_DeflateLevel = 6 ;
current_pixel_state:_NoFill = ""true"" ;
short observation_count(lat, lon) ;
observation_count:standard_name = ""land_cover_lccs number_of_observations"" ;
observation_count:valid_min = 0 ;
observation_count:valid_max = 32767 ;
observation_count:_FillValue = -1s ;
observation_count:long_name = ""number of valid observations"" ;
observation_count:_Storage = ""chunked"" ;
observation_count:_ChunkSizes = 2048, 2048 ;
observation_count:_DeflateLevel = 6 ;
observation_count:_Endianness = ""little"" ;
observation_count:_NoFill = ""true"" ;
byte algorithmic_confidence_level(lat, lon) ;
algorithmic_confidence_level:standard_name = ""land_cover_lccs algorithmic_confidence"" ;
algorithmic_confidence_level:valid_min = 0 ;
algorithmic_confidence_level:valid_max = 100 ;
algorithmic_confidence_level:scale_factor = 0.01f ;
algorithmic_confidence_level:_FillValue = -1b ;
algorithmic_confidence_level:long_name = ""LC map confidence level based on algorithm performance"" ;
algorithmic_confidence_level:_Storage = ""chunked"" ;
algorithmic_confidence_level:_ChunkSizes = 2048, 2048 ;
algorithmic_confidence_level:_DeflateLevel = 6 ;
algorithmic_confidence_level:_NoFill = ""true"" ;
float lat(lat) ;
lat:long_name = ""latitude"" ;
lat:standard_name = ""latitude"" ;
lat:valid_min = -89.9986f ;
lat:valid_max = 89.99861f ;
lat:units = ""degrees_north"" ;
lat:_Storage = ""chunked"" ;
lat:_ChunkSizes = 64800 ;
lat:_DeflateLevel = 6 ;
lat:_Endianness = ""little"" ;
lat:_NoFill = ""true"" ;
float lon(lon) ;
lon:long_name = ""longitude"" ;
lon:standard_name = ""longitude"" ;
lon:valid_min = -179.9986f ;
lon:valid_max = 179.9986f ;
lon:units = ""degrees_east"" ;
lon:_Storage = ""chunked"" ;
lon:_ChunkSizes = 129600 ;
lon:_DeflateLevel = 6 ;
lon:_Endianness = ""little"" ;
lon:_NoFill = ""true"" ;
int crs ;
crs:i2m = ""0.002777777701187,0.0,0.0,-0.002777777701187,-180.00000033927267,90.0"" ;
crs:wkt = ""GEOGCS[\""WGS 84\"", \r\n DATUM[\""World Geodetic System 1984\"", \r\n SPHEROID[\""WGS 84\"", 6378137.0, 298.257223563, AUTHORITY[\""EPSG\"",\""7030\""]], \r\n AUTHORITY[\""EPSG\"",\""6326\""]], \r\n PRIMEM[\""Greenwich\"",
0.0, AUTHORITY[\""EPSG\"",\""8901\""]], \r\n UNIT[\""degree\"", 0.017453292519943295], \r\n AXIS[\""Geodetic longitude\"", EAST], \r\n AXIS[\""Geodetic latitude\"", NORTH], \r\n AUTHORITY[\""EPSG\"",\""4326\""]]"" ;
crs:_Endianness = ""little"" ;
crs:_NoFill = ""true"" ;
// global attributes:
:title = ""ESA CCI Land Cover Map"" ;
:summary = ""This dataset contains the global ESA CCI land cover classification map derived from satellite data of one epoch."" ;
:type = ""ESACCI-LC-L4-LCCS-Map-300m-P5Y"" ;
:id = ""ESACCI-LC-L4-LCCS-Map-300m-P5Y-2005-v1.6.1"" ;
:project = ""Climate Change Initiative - European Space Agency"" ;
:references = ""http://www.esa-landcover-cci.org/"" ;
:institution = ""Universite catholique de Louvain"" ;
:contact = ""landcover-cci@uclouvain.be"" ;
:comment = """" ;
:Conventions = ""CF-1.6"" ;
:standard_name_vocabulary = ""NetCDF Climate and Forecast (CF) Standard Names version 21"" ;
:keywords = ""land cover classification,satellite,observation"" ;
:keywords_vocabulary = ""NASA Global Change Master Directory (GCMD) Science Keywords"" ;
:license = ""ESA CCI Data Policy: free and open access"" ;
:naming_authority = ""org.esa-cci"" ;
:cdm_data_type = ""grid"" ;
:TileSize = ""2048:2048"" ;
:tracking_id = ""00f7e0ee-3b0e-4ea3-9b9f-186e02fb4439"" ;
:product_version = ""1.6.1"" ;
:date_created = ""20151217T094622Z"" ;
:creator_name = ""University catholique de Louvain"" ;
:creator_url = ""http://www.uclouvain.be/"" ;
:creator_email = ""landcover-cci@uclouvain.be"" ;
:source = ""MERIS FR L1B version 5.05, MERIS RR L1B version 8.0, SPOT VGT P"" ;
:history = ""amorgos-4,0, lc-sdr-1.0, lc-sr-1.0, lc-classification-1.0,lc-user-tools-3.10"" ;
:time_coverage_start = ""20030101"" ;
:time_coverage_end = ""20071231"" ;
:time_coverage_duration = ""P5Y"" ;
:time_coverage_resolution = ""P5Y"" ;
:geospatial_lat_min = ""-89.99999"" ;
:geospatial_lat_max = ""90.0"" ;
:geospatial_lon_min = ""-180.0"" ;
:geospatial_lon_max = ""179.99998"" ;
:spatial_resolution = ""300m"" ;
:geospatial_lat_units = ""degrees_north"" ;
:geospatial_lat_resolution = ""0.002778"" ;
:geospatial_lon_units = ""degrees_east"" ;
:geospatial_lon_resolution = ""0.002778"" ;
:_SuperblockVersion = 2 ;
:_IsNetcdf4 = 1 ;
:_Format = ""netCDF-4"" ;
}
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,258500654