home / github / issue_comments

Menu
  • Search all tables
  • GraphQL API

issue_comments: 410773312

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/issues/2304#issuecomment-410773312 https://api.github.com/repos/pydata/xarray/issues/2304 410773312 MDEyOklzc3VlQ29tbWVudDQxMDc3MzMxMg== 1217238 2018-08-06T16:47:11Z 2018-08-06T16:47:22Z MEMBER

A float32 values has 24 bits of precision in the significand, which is more than enough to store the 16-bits in in the original data; the exponent (8 bits) will more or less take care of the * 0.01:

Right. The actual raw data is being stored as an integer 21940 (along with the scale factor of 0.01). Both 21.939998626708984 (as float32) and 21.940000000000001 (as float64) are floating point approximations of the exact decimal number 219.40.

I would be happy to add options for whether to default to float32 or float64 precision. There are clearly tradeoffs here: - float32 uses half the memory - float64 has more precision for downstream computation

I don't think we can make a statement about which is better in general. The best we can do is make an educated guess about which will be more useful / less surprising for most and/or new users, and pick that as the default.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  343659822
Powered by Datasette · Queries took 0.778ms · About: xarray-datasette