issue_comments

Dear all and thank you for your work on Xarray,

Link to @magau comment, I have a netcdf with multiple variables in different format (float, short, byte). Using open_mfdataset 'short' and 'byte' are converted in 'float64' (no scaling, but some masking for the float data). It doesn't raise major issue for me, but it is taking plenty of memory space for nothing.

Below an example of the 3 format from (ncdump -h): short total_nobs(time, lat, lon) ; total_nobs:long_name = "Number of SSS in the time interval" ; total_nobs:valid_min = 0s ; total_nobs:valid_max = 10000s ; float pct_var(time, lat, lon) ; pct_var:_FillValue = NaNf ; pct_var:long_name = "Percentage of SSS_variability that is expected to be not explained by the products" ; pct_var:units = "%" ; pct_var:valid_min = 0. ; pct_var:valid_max = 100. ; byte sss_qc(time, lat, lon) ; sss_qc:long_name = "Sea Surface Salinity Quality, 0=Good; 1=Bad" ; sss_qc:valid_min = 0b ; sss_qc:valid_max = 1b ;

And how they appear after opening in as xarray using open_mfdataset: total_nobs (time, lat, lon) float64 dask.array<chunksize=(48, 584, 1388), meta=np.ndarray> pct_var (time, lat, lon) float32 dask.array<chunksize=(48, 584, 1388), meta=np.ndarray> sss_qc (time, lat, lon) float64 dask.array<chunksize=(48, 584, 1388), met

Is there any recommandation? Regards

Hey everyone, tumbled on this while searching for approximately the same problem. Thought I'd share since the issue is still open. On my part, there is two situations that seem buggy. I haven't been using xarray for that long yet so maybe there is something I'm missing here...

My first problem relates to the data types of dimensions with float notation. To give another answer to @shoyer's question:

To clarify: why is it a problem for you

it is a problem in my case because I would like to perform slicing operations of a dataset using longitude values from another dataset. This operation raises a "KeyError : not all values found in index 'longitude'" since either one of the dataset's longitude is float32 and the other is float64 or because both datasets' float32 approximations are not exactly the same value in each dataset. I can work around this and assign new coords to be float64 after reading and it works, though it is kind of a hassle considering I have to perform this thousands of times. This situation also create a problem when concatenating multiple netCDF files together (along time dim in my case). The discrepancies between the approximations of float32 values or the float32 vs float 64 situation will add new dimension values where it shouldn't.

On the second part of my problem, it comes with writing/reading netCDF files (maybe more related to @daoudjahdou problem). I tried to change the data type to float64 for all my files, save them and then perform what I need to do, but for some reason even though dtype is float64 for all my dimensions when writing the files (using default args), it will sometime be float32, sometime float64 when reading the files (with default ags values) previously saved with float64 dtype. If using the default args, shouldn't the decoding makes the dtype of dimension the same for all files I read?

Hi everyone, I've start using xarray recently, so I apologize if I'm saying something wrong... I've also faced the here reported issue, so have tried to find some answers. Unpacking netcdf files with respect to the NUG attributes (scale_factor and add_offset) seems to be mentioned by the CF-Conventions directives. And it's clear about which data type should be applied to the unpacked data. cf-conventions-1.7/packed-data In this chapter you can read that: "If the scale_factor and add_offset attributes are of the same data type as the associated variable, the unpacked data is assumed to be of the same data type as the packed data. However, if the scale_factor and add_offset attributes are of a different data type from the variable (containing the packed data) then the unpacked data should match the type of these attributes". In my opinion this should be the default behavior of the xarray.decode_cf function. Which doesn't invalidate the idea of forcing the unpacked data dtype. However non of the CFScaleOffsetCoder and CFMaskCoder de/encoder classes seems to be according with these CF directives, since the first one doesn't look for the scale_factor or add_offset dtypes, and the second one also changes the unpacked data dtype (maybe because nan values are being used to replace the fill values). Sorry for such an extensive comment, without any solutions proposal... Regards! :+1:

Hi, thank you for your effort into making xarray a great library. As mentioned in the issue the discussion went on a PR in order to make xr.open_dataset parametrable. This post is about asking you about recommendations regarding our PR.

In this case we would add a parameter to the open_dataset function called "force_promote" which is a boolean and False by default and thus not mandatory. And then spread that parameter down to the function maybe_promote in dtypes.py Where we say the following:

if dtype.itemsize <= 2 and not force_promote: dtype = np.float32 else: dtype = np.float64

The downside of that is that we somehow pollute the code with a parameter that is used in a specific case.

The second approach would check the value of an environment variable called "XARRAY_FORCE_PROMOTE" if it exists and set to true would force promoting type to float64.

please tells us which approach suits best your vision of xarray.

Regards.

Any updates about this ?

As mentioned in the original issue the modification is straightforward. Any ideas if this could be integrated to xarray anytime soon ?

Thank you for your quick answer. In our case we could evaluate std dev or square sums on long lists of values and the accumulation of those small values due to float32 type could create considerable differences.