issue_comments

Table actions

6 rows where issue = 177754433 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

author_association 2

issue 1

  • How to avoid the auto convert variable dtype from float32 to float64 when read netCDF file use open_dataset? · 6 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
377060897 https://github.com/pydata/xarray/issues/1008#issuecomment-377060897 https://api.github.com/repos/pydata/xarray/issues/1008 MDEyOklzc3VlQ29tbWVudDM3NzA2MDg5Nw== shoyer 1217238 2018-03-28T22:37:00Z 2018-03-28T22:37:00Z MEMBER

Yes, this was fixed by @Zac-HD in https://github.com/pydata/xarray/pull/1840

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   How to avoid the auto convert variable dtype from float32 to float64 when read netCDF file use open_dataset? 177754433
376810608 https://github.com/pydata/xarray/issues/1008#issuecomment-376810608 https://api.github.com/repos/pydata/xarray/issues/1008 MDEyOklzc3VlQ29tbWVudDM3NjgxMDYwOA== lvankampenhout 7933853 2018-03-28T08:49:24Z 2018-03-28T08:49:49Z NONE

I stumbled across the same problem in xarray 0.9.1 and updating to 0.10.2 solved it. Perhaps this issue may be closed? 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   How to avoid the auto convert variable dtype from float32 to float64 when read netCDF file use open_dataset? 177754433
248180310 https://github.com/pydata/xarray/issues/1008#issuecomment-248180310 https://api.github.com/repos/pydata/xarray/issues/1008 MDEyOklzc3VlQ29tbWVudDI0ODE4MDMxMA== wqshen 12339722 2016-09-20T01:53:43Z 2016-09-20T01:53:43Z NONE

OK & Thanks

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   How to avoid the auto convert variable dtype from float32 to float64 when read netCDF file use open_dataset? 177754433
248171030 https://github.com/pydata/xarray/issues/1008#issuecomment-248171030 https://api.github.com/repos/pydata/xarray/issues/1008 MDEyOklzc3VlQ29tbWVudDI0ODE3MTAzMA== shoyer 1217238 2016-09-20T00:48:01Z 2016-09-20T00:48:01Z MEMBER

OK, that makes sense. I agree, we could keep such arrays as float32. We don't need to guard against overflow when only decoding a _FillValue.

If you're interested in taking a look at a fix, this is where the current logic is: https://github.com/pydata/xarray/blob/551a7bca7b42a6d6db976b01d2eee1131735785d/xarray/conventions.py#L795-L802

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   How to avoid the auto convert variable dtype from float32 to float64 when read netCDF file use open_dataset? 177754433
248170532 https://github.com/pydata/xarray/issues/1008#issuecomment-248170532 https://api.github.com/repos/pydata/xarray/issues/1008 MDEyOklzc3VlQ29tbWVudDI0ODE3MDUzMg== wqshen 12339722 2016-09-20T00:44:27Z 2016-09-20T00:44:27Z NONE

Thanks for your reply.

Variables dbz, vr and sw have the _FillValue attribute, whose value is equal to _FillValue: -999.0

<type 'netCDF4._netCDF4.Variable'> float32 dbz(z, y, x) _FillValue: -999.0 units: dBZ long_name: reflectivity in log units unlimited dimensions: current shape = (9, 461, 461) filling on

It seems that xarray will convert the dtype from float32 to float64, while the variable in netCDF4 file has the attribute _FillValue, xr.open_dataset auto identify the _FillValue in the variable and mask it with np.nan , however the variable is also changed into float64. I think this result with default argument is not reasonable enough, as this type conversion is not necessary in fact.

Using mask_and_scale=False will maintain the variable in float32 and retain the _FillValue attribute of variable. However, what i want is mask but maintain the dtype in float32. Is it has possible bug in the internal of the open_dataset method ?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   How to avoid the auto convert variable dtype from float32 to float64 when read netCDF file use open_dataset? 177754433
248041752 https://github.com/pydata/xarray/issues/1008#issuecomment-248041752 https://api.github.com/repos/pydata/xarray/issues/1008 MDEyOklzc3VlQ29tbWVudDI0ODA0MTc1Mg== shoyer 1217238 2016-09-19T16:22:02Z 2016-09-19T16:22:02Z MEMBER

Some of your variables probably have scale_factor or add_offset attributes. To avoid overflow, xarray converts such variables to float64 when scaling.

You can disable the scaling by writing xr.open_dataset('cat.20151003200633.nc', mask_and_scale=False)

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   How to avoid the auto convert variable dtype from float32 to float64 when read netCDF file use open_dataset? 177754433

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);