issues

Table actions

2 rows where repo = 13221727, type = "issue" and user = 90768774 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date), closed_at (date)

state 2

type 1

  • issue · 2 ✖

repo 1

  • xarray · 2 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
1841227199 I_kwDOAMm_X85tvuW_ 8055 Documentation of `Dataset.count()` is ambiguous Articoking 90768774 closed 0     3 2023-08-08T12:35:59Z 2023-08-09T13:50:56Z 2023-08-09T13:50:56Z CONTRIBUTOR      

What is your issue?

Hello everyone!

I've noticed that the documentation for Dataset.count and DataArray.count is a little unclear with respect to what it actually does. In my experience, it's not immediately obvious when looking at this site whether zero values are counted or not.

The given example does not contain a 0 value that would clarify this, and what is worse is that the "see also" section points to np.count and to dask.array.count which do not exist. The closest NumPy equivalent is np.count_nonzero, which has a different behavior to the xarray count method. The xarray count is actually equivalent to Pandas DataFrame.count, which count all non-NA values including 0.

I would take a shot at writing a clearer docstring but I'm not sure how to do that using the generate_aggregations.py file...

 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/8055/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed xarray 13221727 issue
1681353195 I_kwDOAMm_X85kN2nr 7782 xr.open_dataset() reading ubyte variables as float32 from DAP server Articoking 90768774 open 0     12 2023-04-24T14:04:13Z 2023-04-27T13:33:50Z   CONTRIBUTOR      

What happened?

Trying to open and save a netcdf file through CEDA's DAP server (http://dap.ceda.ac.uk/thredds/dodsC/neodc/esacci/snow/data/scfv/MODIS/v2.0/2010/01/20100101-ESACCI-L3C_SNOW-SCFV-MODIS_TERRA-fv2.0.nc) whose variables scfv and scfv_unc are of type ubyte. File DDS is as follows:

Dataset { Float64 lat_bnds[lat = 18000][nv = 2]; Float64 lon_bnds[lon = 36000][nv = 2]; Float32 time[time = 1]; Float64 lat[lat = 18000]; Float64 lon[lon = 36000]; Grid { ARRAY: Byte scfv[time = 1][lat = 18000][lon = 36000]; MAPS: Float32 time[time = 1]; Float64 lat[lat = 18000]; Float64 lon[lon = 36000]; } scfv; Grid { ARRAY: Byte scfv_unc[time = 1][lat = 18000][lon = 36000]; MAPS: Float32 time[time = 1]; Float64 lat[lat = 18000]; Float64 lon[lon = 36000]; } scfv_unc; Int32 spatial_ref; } neodc/esacci/snow/data/scfv/MODIS/v2.0/2010/01/20100101-ESACCI-L3C_SNOW-SCFV-MODIS_TERRA-fv2.0.nc;

And its DAS has attribute _Unsigned as true.

Attributes { lat_bnds { } lon_bnds { } time { String axis "T"; String standard_name "time"; String long_name "time"; String calendar "standard"; String units "hours since 1950-01-01 00:00:00"; Int32 _ChunkSizes 1024; } lat { String axis "Y"; String standard_name "latitude"; String long_name "WGS84 latitude coordinates, center of pixel"; String units "degrees_north"; Float64 actual_range -89.995, 89.995; String bounds "lat_bnds"; Int32 _ChunkSizes 18000; } lon { String axis "X"; String standard_name "longitude"; String long_name "WGS84 longitude coordinates, center of pixel"; String units "degrees_east"; Float64 actual_range -179.995, 179.995; String bounds "lon_bnds"; Int32 _ChunkSizes 36000; } scfv { String _Unsigned "true"; Int16 _FillValue -1; String standard_name "snow_area_fraction_viewable_from_above"; String long_name "Snow Cover Fraction Viewable"; String units "percent"; Int16 valid_range 0, -2; Byte actual_range 0, 100; Int16 flag_values -51, -50, -46, -41, -4, -3, -2; String flag_meanings "Cloud Polar_Night_or_Night Water Permanent_Snow_and_Ice Classification_failed Input_Data_Error No_Satellite_Acquisition"; Int16 missing_value -1; String ancillary_variables "scfv_unc"; String grid_mapping "spatial_ref"; Int32 _ChunkSizes 1, 1385, 2770; } scfv_unc { String _Unsigned "true"; Int16 _FillValue -1; String standard_name "snow_area_fraction_viewable_from_above standard_error"; String long_name "Unbiased Root Mean Square Error for Snow Cover Fraction Viewable"; String units "percent"; Int16 valid_range 0, -2; Byte actual_range 0, 100; Int16 flag_values -51, -50, -46, -41, -4, -3, -2; String flag_meanings "Cloud Polar_Night_or_Night Water Permanent_Snow_and_Ice Classification_failed Input_Data_Error No_Satellite_Acquisition"; Int16 missing_value -1; String grid_mapping "spatial_ref"; Int32 _ChunkSizes 1, 1385, 2770; } ...

Using xr.open_dataset(http://dap.ceda.ac.uk/thredds/dodsC/neodc/esacci/snow/data/scfv/MODIS/v2.0/2010/01/20100101-ESACCI-L3C_SNOW-SCFV-MODIS_TERRA-fv2.0.nc) the mentioned variables get read as float32 instead of ubyte or at least byte

What did you expect to happen?

The returned Dataset should have scfv and scfv_unc of dtype ubyte

Minimal Complete Verifiable Example

Python import xarray as xr ds = xr.open_dataset("http://dap.ceda.ac.uk/thredds/dodsC/neodc/esacci/snow/data/scfv/MODIS/v2.0/2010/01/20100101-ESACCI-L3C_SNOW-SCFV-MODIS_TERRA-fv2.0.nc") print(ds)

MVCE confirmation

  • [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • [X] Complete example — the example is self-contained, including all data and the text of any traceback.
  • [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

Python /usr/local/install/python-3.9/lib/python3.9/site-packages/xarray/conventions.py:523: SerializationWarning: variable 'scfv' has multiple fill values {-1, 255}, decoding all values to NaN. new_vars[k] = decode_cf_variable( /usr/local/install/python-3.9/lib/python3.9/site-packages/xarray/conventions.py:523: SerializationWarning: variable 'scfv_unc' has multiple fill values {-1, 255}, decoding all values to NaN. new_vars[k] = decode_cf_variable( <xarray.Dataset> Dimensions: (lat: 18000, nv: 2, lon: 36000, time: 1) Coordinates: * time (time) datetime64[ns] 2010-01-01 * lat (lat) float64 -90.0 -89.98 -89.97 -89.97 ... 89.98 89.98 90.0 * lon (lon) float64 -180.0 -180.0 -180.0 -180.0 ... 180.0 180.0 180.0 Dimensions without coordinates: nv Data variables: lat_bnds (lat, nv) float64 ... lon_bnds (lon, nv) float64 ... spatial_ref int32 ... scfv (time, lat, lon) float32 ... scfv_unc (time, lat, lon) float32 ... Attributes: (12/43) title: ESA CCI viewable snow product level L3C ... institution: ENVEO IT GmbH source: TERRA MODIS, Collection 6.1: calibrated ... history: 2021-12-06: ESA snow_cci processing line... references: http://snow-cci.enveo.at/ tracking_id: 2be7fb10-d660-497c-99b9-086372ec6c83 ... ... platform: TERRA sensor: MODIS spatial_resolution: 0.01 degree key_variables: scfv doi: 10.5285/ebe625b6f77945a68bda0ab7c78dd76b DODS_EXTRA.Unlimited_Dimension: time

Anything else we need to know?

No response

Environment

INSTALLED VERSIONS ------------------ commit: None python: 3.9.15 (main, Nov 24 2022, 14:31:59) [GCC 11.2.0] python-bits: 64 OS: Linux OS-release: 3.10.0-1160.80.1.el7.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.12.2 libnetcdf: 4.8.1 xarray: 2022.11.0 pandas: 1.5.2 numpy: 1.23.5 scipy: 1.9.3 netCDF4: 1.6.2 pydap: None h5netcdf: None h5py: 3.7.0 Nio: None zarr: None cftime: 1.6.2 nc_time_axis: None PseudoNetCDF: None rasterio: 1.3.3 cfgrib: 0.9.10.3 iris: None bottleneck: None dask: 2022.11.1 distributed: None matplotlib: 3.5.2 cartopy: 0.21.0 seaborn: 0.12.1 numbagg: None fsspec: 2022.11.0 cupy: None pint: 0.20.1 sparse: 0.13.0 flox: None numpy_groupies: None setuptools: 65.5.0 pip: 22.2.2 conda: 22.9.0 pytest: 7.2.0 IPython: 7.33.0 sphinx: 5.3.0 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/7782/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   reopened xarray 13221727 issue

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [active_lock_reason] TEXT,
   [draft] INTEGER,
   [pull_request] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [state_reason] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
);
CREATE INDEX [idx_issues_repo]
    ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
    ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
    ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
    ON [issues] ([user]);