html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/5597#issuecomment-1492937244,https://api.github.com/repos/pydata/xarray/issues/5597,1492937244,IC_kwDOAMm_X85Y_Goc,5821660,2023-04-01T11:03:02Z,2023-04-01T11:03:02Z,MEMBER,"
> To fix this, I think logic in `_choose_float_dtype` should be updated to look at `encoding['dtype']` (if available) instead of `dtype`, in order to understand how the data was originally stored.
This is aimed at in #7654
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,942738904
https://github.com/pydata/xarray/issues/5597#issuecomment-879561954,https://api.github.com/repos/pydata/xarray/issues/5597,879561954,MDEyOklzc3VlQ29tbWVudDg3OTU2MTk1NA==,1217238,2021-07-14T03:43:37Z,2021-07-14T03:44:00Z,MEMBER,"Thanks for sharing the subset netCDF file, that is very helpful for debugging indeed!
The weird thing is that the dtype picking logic _seems_ to have a special case that, per the code comment, suggesting we want to be using float64 here:
https://github.com/pydata/xarray/blob/eea76733770be03e78a0834803291659136bca31/xarray/coding/variables.py#L231-L238
But in fact, the dtype picking logic doesn't do that, because the dtype is already converted into float32, first. The culprit seems to be this line in CFMaskCoder, which promotes the dtype to float32 to fit a fill-value of NaN:
https://github.com/pydata/xarray/blob/eea76733770be03e78a0834803291659136bca31/xarray/coding/variables.py#L202
To fix this, I think logic in `_choose_float_dtype` should be updated to look at `encoding['dtype']` (if available) instead of `dtype`, in order to understand how the data was originally stored.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,942738904
https://github.com/pydata/xarray/issues/5597#issuecomment-879361320,https://api.github.com/repos/pydata/xarray/issues/5597,879361320,MDEyOklzc3VlQ29tbWVudDg3OTM2MTMyMA==,1217238,2021-07-13T19:58:39Z,2021-07-13T19:58:39Z,MEMBER,"This may just be the expected floating point error from using float32:
```
In [5]: import numpy as np
In [6]: -32766 * np.float32(625.6492454183389) + np.float32(20500023.17537729)
Out[6]: 1.2984619140625
```
If you use full float64, then the data does decode to 0.0:
```
In [7]: -32766 * np.float64(625.6492454183389) + np.float64(20500023.17537729)
Out[7]: 0.0
```
So the question then is why this ends up being decoded using float32 instead of float64, and if that logic should be adjusted or made customizable:
https://github.com/pydata/xarray/blob/eea76733770be03e78a0834803291659136bca31/xarray/coding/variables.py#L225","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,942738904
https://github.com/pydata/xarray/issues/5597#issuecomment-879199563,https://api.github.com/repos/pydata/xarray/issues/5597,879199563,MDEyOklzc3VlQ29tbWVudDg3OTE5OTU2Mw==,14808389,2021-07-13T15:45:05Z,2021-07-13T15:45:05Z,MEMBER,"can you post the value of `xarray.open_dataset(""BIG_FILE_packed.nc"").ssrd.encoding`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,942738904
https://github.com/pydata/xarray/issues/5597#issuecomment-878914016,https://api.github.com/repos/pydata/xarray/issues/5597,878914016,MDEyOklzc3VlQ29tbWVudDg3ODkxNDAxNg==,5821660,2021-07-13T09:07:14Z,2021-07-13T09:07:14Z,MEMBER,"@ohsqueezy You might also try `engine=""h5netcdf` (`h5py`/`h5netcdf` packages needed). And would it be possible create a small subset of that file via netCDF4 to share?","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,942738904
https://github.com/pydata/xarray/issues/5597#issuecomment-878833159,https://api.github.com/repos/pydata/xarray/issues/5597,878833159,MDEyOklzc3VlQ29tbWVudDg3ODgzMzE1OQ==,5635139,2021-07-13T07:03:43Z,2021-07-13T07:03:43Z,MEMBER,Thanks. Does passing different values to `engine=` make any difference? I suspect the issue is at that layer.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,942738904
https://github.com/pydata/xarray/issues/5597#issuecomment-878830308,https://api.github.com/repos/pydata/xarray/issues/5597,878830308,MDEyOklzc3VlQ29tbWVudDg3ODgzMDMwOA==,5821660,2021-07-13T06:58:24Z,2021-07-13T06:58:24Z,MEMBER,"@ohsqueezy Does this issue also show up, if just plain netCDF4 is used to open the files?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,942738904
https://github.com/pydata/xarray/issues/5597#issuecomment-878792388,https://api.github.com/repos/pydata/xarray/issues/5597,878792388,MDEyOklzc3VlQ29tbWVudDg3ODc5MjM4OA==,5635139,2021-07-13T05:32:27Z,2021-07-13T05:32:27Z,MEMBER,"A small question to clarify:
>
> When the netCDF is unpacked using the nco command line tool, the correct values are unpacked.
> ```python
> $ xarray.open_dataset(""BIG_FILE_unpacked.nc"").ssrd.isel(time=slice(0, 23)).sel(latitude=44.8, longitude=287.1, method=""nearest"").values
> > array([ 0. , 0. , 0. ,
> 0. , 0. , 0. ,
> 0. , 0. , 0. ,
> 0. , 0. , 0. ,
> 25651.61906215, 354743.1221522 , 1091757.933255 ,
> 2170377.23235622, 3482363.69999847, 4704882.32554591,
> 5689654.23783437, 6297785.304381 , 6534906.36839455,
> 6543665.4578304 , 6543665.4578304 ])
> ```
Is that the output of the `.open_dataset` command? Is that the same code that generates the 2s? Or is the command that generates the zero different?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,942738904