home / github / issues

Menu
  • GraphQL API
  • Search all tables

issues: 1681353195

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
1681353195 I_kwDOAMm_X85kN2nr 7782 xr.open_dataset() reading ubyte variables as float32 from DAP server 90768774 open 0     12 2023-04-24T14:04:13Z 2023-04-27T13:33:50Z   CONTRIBUTOR      

What happened?

Trying to open and save a netcdf file through CEDA's DAP server (http://dap.ceda.ac.uk/thredds/dodsC/neodc/esacci/snow/data/scfv/MODIS/v2.0/2010/01/20100101-ESACCI-L3C_SNOW-SCFV-MODIS_TERRA-fv2.0.nc) whose variables scfv and scfv_unc are of type ubyte. File DDS is as follows:

Dataset { Float64 lat_bnds[lat = 18000][nv = 2]; Float64 lon_bnds[lon = 36000][nv = 2]; Float32 time[time = 1]; Float64 lat[lat = 18000]; Float64 lon[lon = 36000]; Grid { ARRAY: Byte scfv[time = 1][lat = 18000][lon = 36000]; MAPS: Float32 time[time = 1]; Float64 lat[lat = 18000]; Float64 lon[lon = 36000]; } scfv; Grid { ARRAY: Byte scfv_unc[time = 1][lat = 18000][lon = 36000]; MAPS: Float32 time[time = 1]; Float64 lat[lat = 18000]; Float64 lon[lon = 36000]; } scfv_unc; Int32 spatial_ref; } neodc/esacci/snow/data/scfv/MODIS/v2.0/2010/01/20100101-ESACCI-L3C_SNOW-SCFV-MODIS_TERRA-fv2.0.nc;

And its DAS has attribute _Unsigned as true.

Attributes { lat_bnds { } lon_bnds { } time { String axis "T"; String standard_name "time"; String long_name "time"; String calendar "standard"; String units "hours since 1950-01-01 00:00:00"; Int32 _ChunkSizes 1024; } lat { String axis "Y"; String standard_name "latitude"; String long_name "WGS84 latitude coordinates, center of pixel"; String units "degrees_north"; Float64 actual_range -89.995, 89.995; String bounds "lat_bnds"; Int32 _ChunkSizes 18000; } lon { String axis "X"; String standard_name "longitude"; String long_name "WGS84 longitude coordinates, center of pixel"; String units "degrees_east"; Float64 actual_range -179.995, 179.995; String bounds "lon_bnds"; Int32 _ChunkSizes 36000; } scfv { String _Unsigned "true"; Int16 _FillValue -1; String standard_name "snow_area_fraction_viewable_from_above"; String long_name "Snow Cover Fraction Viewable"; String units "percent"; Int16 valid_range 0, -2; Byte actual_range 0, 100; Int16 flag_values -51, -50, -46, -41, -4, -3, -2; String flag_meanings "Cloud Polar_Night_or_Night Water Permanent_Snow_and_Ice Classification_failed Input_Data_Error No_Satellite_Acquisition"; Int16 missing_value -1; String ancillary_variables "scfv_unc"; String grid_mapping "spatial_ref"; Int32 _ChunkSizes 1, 1385, 2770; } scfv_unc { String _Unsigned "true"; Int16 _FillValue -1; String standard_name "snow_area_fraction_viewable_from_above standard_error"; String long_name "Unbiased Root Mean Square Error for Snow Cover Fraction Viewable"; String units "percent"; Int16 valid_range 0, -2; Byte actual_range 0, 100; Int16 flag_values -51, -50, -46, -41, -4, -3, -2; String flag_meanings "Cloud Polar_Night_or_Night Water Permanent_Snow_and_Ice Classification_failed Input_Data_Error No_Satellite_Acquisition"; Int16 missing_value -1; String grid_mapping "spatial_ref"; Int32 _ChunkSizes 1, 1385, 2770; } ...

Using xr.open_dataset(http://dap.ceda.ac.uk/thredds/dodsC/neodc/esacci/snow/data/scfv/MODIS/v2.0/2010/01/20100101-ESACCI-L3C_SNOW-SCFV-MODIS_TERRA-fv2.0.nc) the mentioned variables get read as float32 instead of ubyte or at least byte

What did you expect to happen?

The returned Dataset should have scfv and scfv_unc of dtype ubyte

Minimal Complete Verifiable Example

Python import xarray as xr ds = xr.open_dataset("http://dap.ceda.ac.uk/thredds/dodsC/neodc/esacci/snow/data/scfv/MODIS/v2.0/2010/01/20100101-ESACCI-L3C_SNOW-SCFV-MODIS_TERRA-fv2.0.nc") print(ds)

MVCE confirmation

  • [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • [X] Complete example — the example is self-contained, including all data and the text of any traceback.
  • [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

Python /usr/local/install/python-3.9/lib/python3.9/site-packages/xarray/conventions.py:523: SerializationWarning: variable 'scfv' has multiple fill values {-1, 255}, decoding all values to NaN. new_vars[k] = decode_cf_variable( /usr/local/install/python-3.9/lib/python3.9/site-packages/xarray/conventions.py:523: SerializationWarning: variable 'scfv_unc' has multiple fill values {-1, 255}, decoding all values to NaN. new_vars[k] = decode_cf_variable( <xarray.Dataset> Dimensions: (lat: 18000, nv: 2, lon: 36000, time: 1) Coordinates: * time (time) datetime64[ns] 2010-01-01 * lat (lat) float64 -90.0 -89.98 -89.97 -89.97 ... 89.98 89.98 90.0 * lon (lon) float64 -180.0 -180.0 -180.0 -180.0 ... 180.0 180.0 180.0 Dimensions without coordinates: nv Data variables: lat_bnds (lat, nv) float64 ... lon_bnds (lon, nv) float64 ... spatial_ref int32 ... scfv (time, lat, lon) float32 ... scfv_unc (time, lat, lon) float32 ... Attributes: (12/43) title: ESA CCI viewable snow product level L3C ... institution: ENVEO IT GmbH source: TERRA MODIS, Collection 6.1: calibrated ... history: 2021-12-06: ESA snow_cci processing line... references: http://snow-cci.enveo.at/ tracking_id: 2be7fb10-d660-497c-99b9-086372ec6c83 ... ... platform: TERRA sensor: MODIS spatial_resolution: 0.01 degree key_variables: scfv doi: 10.5285/ebe625b6f77945a68bda0ab7c78dd76b DODS_EXTRA.Unlimited_Dimension: time

Anything else we need to know?

No response

Environment

INSTALLED VERSIONS ------------------ commit: None python: 3.9.15 (main, Nov 24 2022, 14:31:59) [GCC 11.2.0] python-bits: 64 OS: Linux OS-release: 3.10.0-1160.80.1.el7.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.12.2 libnetcdf: 4.8.1 xarray: 2022.11.0 pandas: 1.5.2 numpy: 1.23.5 scipy: 1.9.3 netCDF4: 1.6.2 pydap: None h5netcdf: None h5py: 3.7.0 Nio: None zarr: None cftime: 1.6.2 nc_time_axis: None PseudoNetCDF: None rasterio: 1.3.3 cfgrib: 0.9.10.3 iris: None bottleneck: None dask: 2022.11.1 distributed: None matplotlib: 3.5.2 cartopy: 0.21.0 seaborn: 0.12.1 numbagg: None fsspec: 2022.11.0 cupy: None pint: 0.20.1 sparse: 0.13.0 flox: None numpy_groupies: None setuptools: 65.5.0 pip: 22.2.2 conda: 22.9.0 pytest: 7.2.0 IPython: 7.33.0 sphinx: 5.3.0
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/7782/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  reopened 13221727 issue

Links from other tables

  • 2 rows from issues_id in issues_labels
  • 12 rows from issue in issue_comments
Powered by Datasette · Queries took 0.819ms · About: xarray-datasette