issue_comments

I'm dropping a quick note here to flag that PR #1453 has been written to address this issue. I have used the draft PR in earnest as part of some ongoing analyses of the data that led me to raise the issue.

Ah, I see the distinction concerning the xarray implementation being made in the earlier discussion. There certainly are some tradeoffs being made by the data provider, and I'm far enough removed from those decisions that there's not much I can do.

Thanks for the clarification on the background discussion and the willingness to consider support for the attribute. It's my first time dealing with xarray internals, but given that there's a reference implementation in NetCDF4-python, I'm willing to have a go at a PR for this.

If I had to take a guess, it belongs in conventions.py, fitting with the logic beginning line 789 where the mask_and_scale is handled. It could be built into the MaskedAndScaledArray class, though perhaps a new UnsignedIntArray class might be preferred?

The docstrings note that the implementation is designed around CF-style data. As noted above, this attribute is outside the CF conventions, so I might note that exception in a comment in the code.

HDF5 doesn't come without its own draw backs...such as vastly more complicated (and admittedly more capable) on-disk format.

Yes, of course. But I thought netCDF4-extended meant NetCDF on HDF5, making use of all the HD5 features. So I didn't think that was the tradeoff here.

Also, while unsigned types are a feature of the netCDF4/HDF5, strictly speaking, the netCDF4 unsigned types are not compatible with any released version of the CF specification

Sure, though if you're strictly following CF, you probably would use valid_min/valid_max/valid_range instead of _Unsigned...

Anyways, again I don't really have an objection here. This attribute can be interpreted in a pretty unambiguous way, so adding support for this would be mostly harmless.

HDF5 doesn't come without its own draw backs...such as vastly more complicated (and admittedly more capable) on-disk format. We should not presume to know better than the data providers what their trade-offs are.

Also, while unsigned types are a feature of the netCDF4/HDF5, strictly speaking, the netCDF4 unsigned types are not compatible with any released version of the CF specification (see list of acceptable data types here: http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html#_data_types).

So it's not exactly as simple as you'd like it to be.

To clarify: I didn't want to let NetCDF4-python parse this attribute, because it would break an invariant in our data model. I'm not opposed to supporting this separately in xarray (using xarray's machinery for handling netcdf conventions).

I will say though that this convention makes little sense to me for netcdf4/hdf5 files, which already provide native support for unsigned types, so separately I would also encourage this data provider to change their practices here :).