issue_comments

I can try to fix this in a PR, I just need to be sure what the fix should look like - to change the dimensionality of attributes (has the potential to break backward compatibility) or to adapt other components to handle either scalars or length 1 arrays (safer alternative, but may occur in more locations both inside and outside xarray, so in this case perhaps a note in the documentation could be in order as well). I don't know if xarray thrives for consistency between what the different engines expose on opening the same file.

According to The NetCDF User's Guide, attributes are supposed to be vectors:

The current version treats all attributes as vectors; scalar values are treated as single-element vectors.

That suggests that, strictly speaking, the h5netcdf engine is right and the netcdf4 engine is wrong, and that other components (such as where the scale factor and add_offset are applied) need to be adapted to handle arrays of length 1 for those values.

The ds.load() prevents the traceback because it means the entire n-d data variable is multiplied with the 1-d scale factor. Similarly, requesting a slice (ds["Rad"][400:402, 300:302]) also prevents a traceback. The traceback occurs if a single value is requested, because then Python will complain about multiplying a scalar with a 1-d array. I'm not entirely sure why, but would be a numpy issue:

``` In [7]: a = np.array(0)

In [8]: b = np.array([0])

In [9]: a * b Out[9]: array([0])

In [10]: a *= b

ValueError Traceback (most recent call last) <ipython-input-10-0d04f348f081> in <module> ----> 1 a *= b

ValueError: non-broadcastable output operand with shape () doesn't match the broadcast shape (1,) ```

Interestingly, the problem is prevented if one adds

ds.load()

before the print statement.

My last comment was inaccurate. Although the open succeeds, the non-scalar scale factor does trigger failure upon accessing data (due to lazy loading) even without any open file:

python import xarray fn = "OR_ABI-L1b-RadF-M3C07_G16_s20170732006100_e20170732016478_c20170732016514.nc" with xarray.open_dataset(fn, engine="h5netcdf") as ds: print(ds["Rad"][400, 300])

The data file is publicly available at:

s3://noaa-goes16/ABI-L1b-RadF/2017/073/20/OR_ABI-L1b-RadF-M3C07_G16_s20170732006100_e20170732016478_c20170732016514.nc

However, a simple `xarray.open_dataset(fn, engine="h5netcdf") still fails with ValueError only if passed an open file, so there appear to be still other differences apart from the dimensionality of the variable attributes depending on backend.

Some further digging shows it's due to differences between the h5netcdf and netcdf4 backends:

python import xarray fn = "/data/gholl/cache/fogtools/abi/2017/03/14/20/06/7/OR_ABI-L1b-RadF-M3C07_G16_s20170732006100_e20170732016478_c20170732016514.nc" with xarray.open_dataset(fn, decode_cf=False, mask_and_scale=False, engine="netcdf4") as ds: print(ds["esun"].attrs["_FillValue"]) print(ds["Rad"].attrs["scale_factor"]) with xarray.open_dataset(fn, decode_cf=False, mask_and_scale=False, engine="h5netcdf") as ds: print(ds["esun"].attrs["_FillValue"]) print(ds["Rad"].attrs["scale_factor"])

Results in:

-999.0 0.001564351 [-999.] [0.00156435]

Probably related: when reading an open file through a file system instance, the _FillValue, scale_factor, and add_offset are arrays of length one. When opening by passing a filename, those are all scalar (as expected):

python import xarray from fsspec.implementations.local import LocalFileSystem fn = "/data/gholl/cache/fogtools/abi/2017/03/14/20/06/7/OR_ABI-L1b-RadF-M3C07_G16_s20170732006100_e20170732016478_c20170732016514.nc" ds1 = xarray.open_dataset(fn, decode_cf=True, mask_and_scale=False) print(ds1["esun"].attrs["_FillValue"]) print(ds1["Rad"].attrs["scale_factor"]) with LocalFileSystem().open(fn) as of: ds2 = xarray.open_dataset(of, decode_cf=True, mask_and_scale=False) print(ds2["esun"].attrs["_FillValue"]) print(ds2["Rad"].attrs["scale_factor"])

Result:

-999.0 0.001564351 [-999.] [0.00156435]

I strongly suspect that this is what causes the ValueError, and in any case it also causes downstream problems even if opening succeeds as per the previous comment.

I just tested this with some more combinations:

• decode_cf=True, mask_and_scale=False, everything seems fine.
• decode_cf=False, mask_and_scale=True, everything seems fine.
• decode_cf=True, mask_and_scale=True results in the ValueError and associated traceback.