issue_comments

Hi @etsmith14. The suggestion I did loses accuracy and depending on the variable this is not acceptable. However, recomputing scale_factor and add_offset is possible: https://github.com/ArcticSnow/TopoPyScale/issues/60#issuecomment-1460022033 It is more complicated than dropping the dtype, but it does keep the filesize small.

@etsmith14: another workaround is removing the scale_factor instead of the dtype. This keeps the file size small. However, there are slight offsets between the source and destination datasets, which is to be expected since the original value for the msl variable was in the range of 0.1/0.11 and removing it defaults to 1. For your variable, the scale_factor might also be completely different. However, maybe the scale_factor (and add_offset) can be replaced by something that works for all ERA5 data instead of a value very specific to a single dataset/period.

I have been thinking about a desireable solution, but I have a bit of trouble with it. Besides removing dtype from encoding (resulting in floats being written), one could also change the scale_factor to a higher value (e.g. 0.5). Writing this to int does take half the disksize than releasing the int restriction and writing it to float32. Whatever you do, the data is altered at least slightly.

Apparently, the data cannot be properly written to integers after reading it. This is a bit odd I would say, would that mean that the scaling+offset of ERA5 data is that thightly chosen that when applying it to another dataset/month, the data would fall out of the integer reach? Would be great if this would "just work". At the moment, apparently reading and writing ERA5 data with xarray results in incorrect netcdf files. I expected xarray would work off the shelf with these type of data, it feels like xarray is designed for doing exactly these type of things.

I have also encountered an issue with reading of ERA5 data with open_mfdatset, writing it to_netcdf() and reading it again (https://github.com/Deltares/dfm_tools/issues/239). I was actually looking for a place to land this, and found your issue.

My expectation is that this is because the ERA5 data is saved as ints, but all files have different offsets/scalingfactors. Upon opening it with open_mfdataset(), the data is converted to floats and to the offset/scalingfactor of the first file. This is fine, but the issue occurs I think (and what you also mention) since {'dtype': 'int16'} is in the encoding. The file is written as ints and this seems to mess up the data. (all a theory)

A workaround is to remove the dtype from the encoding for all variables in the file (or update to float32), but that seems cumbersome.