issue_comments

I am closing this. Feel free to reopen if you think there more we can do.

I don't really see through. The

python-traceback UFuncTypeError: ufunc 'multiply' did not contain a loop with signature matching types (dtype('<U6'), dtype('<U6')) -> dtype('<U6') error hints at a dtype problem. The B01:scale_factor = "0.0001" has dtype=<U6 so maybe it fails trying to apply the scale_factor. The other error python traceback RuntimeError: NetCDF: Attempting netcdf-4 operation on netcdf-3 file sounds like a netCDF3 file is pretending to be a netCDF4 file. Also file HLS.S30.T13SDD.2020004.v1.4.hdf tells me it's a hdf4 file. I thought netCDF4 files have to be hdf5 (but I am not sure).

To summarize 1. the file is maybe not meant to be opened as a netCDF but with a hdf library 2. the file may have an issue, either with the metadata or in the way the file is saved 3. if 2. and 3. are not the case, there may be a bug in the netCDF4 library (because you also get the error using netCDF4 without xarray) So I am tagging this as upstream issue.

thanks for your reply, @mathause !

xr.open_dataset(f, decode_cf=False) returns the following traceback with xarray 0.16.1, but providing that keyword in version 0.14.1 seems to have loaded the data but at the cost of some metadata like nodata and coordinates on the axes. But maybe that data isn't structured such that is easily decoded?

```

RuntimeError Traceback (most recent call last) <ipython-input-34-e087a0866e6e> in <module> 1 # ds = xr.open_dataset(f.replace('.hdf', '.h4')) ----> 2 ds = xr.open_dataset(f, decode_cf=False) 3 ds['B01'].values

C:\software\Anaconda3\envs\x-python\lib\site-packages\xarray\backends\api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs, use_cftime, decode_timedelta) 543 544 with close_on_error(store): --> 545 ds = maybe_decode_store(store) 546 547 # Ensure source filename always stored in dataset object (GH issue #2550)

C:\software\Anaconda3\envs\x-python\lib\site-packages\xarray\backends\api.py in maybe_decode_store(store, lock) 457 drop_variables=drop_variables, 458 use_cftime=use_cftime, --> 459 decode_timedelta=decode_timedelta, 460 ) 461

C:\software\Anaconda3\envs\x-python\lib\site-packages\xarray\conventions.py in decode_cf(obj, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta) 580 encoding = obj.encoding 581 elif isinstance(obj, AbstractDataStore): --> 582 vars, attrs = obj.load() 583 extra_coords = set() 584 file_obj = obj

C:\software\Anaconda3\envs\x-python\lib\site-packages\xarray\backends\common.py in load(self) 110 """ 111 variables = FrozenDict( --> 112 (_decode_variable_name(k), v) for k, v in self.get_variables().items() 113 ) 114 attributes = FrozenDict(self.get_attrs())

C:\software\Anaconda3\envs\x-python\lib\site-packages\xarray\backends\netCDF4_.py in get_variables(self) 398 def get_variables(self): 399 dsvars = FrozenDict( --> 400 (k, self.open_store_variable(k, v)) for k, v in self.ds.variables.items() 401 ) 402 return dsvars

C:\software\Anaconda3\envs\x-python\lib\site-packages\xarray\core\utils.py in FrozenDict(args, kwargs) 440 441 def FrozenDict(args, kwargs) -> Frozen: --> 442 return Frozen(dict(*args, kwargs)) 443 444

C:\software\Anaconda3\envs\x-python\lib\site-packages\xarray\backends\netCDF4_.py in <genexpr>(.0) 398 def get_variables(self): 399 dsvars = FrozenDict( --> 400 (k, self.open_store_variable(k, v)) for k, v in self.ds.variables.items() 401 ) 402 return dsvars

C:\software\Anaconda3\envs\x-python\lib\site-packages\xarray\backends\netCDF4_.py in open_store_variable(self, name, var) 374 # netCDF4 specific encoding; save _FillValue for later 375 encoding = {} --> 376 filters = var.filters() 377 if filters is not None: 378 encoding.update(filters)

netCDF4_netCDF4.pyx in netCDF4._netCDF4.Variable.filters()

netCDF4_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()

RuntimeError: NetCDF: Attempting netcdf-4 operation on netcdf-3 file ``ncdump -k HLS.S30.T13SDD.2020004.v1.4.hdfreturnsnetCDF-4`

ncdump -h HLS.S30.T13SDD.2020004.v1.4.hdf returns

``` netcdf HLS.S30.T13SDD.2020004.v1.4 { dimensions: YDim_Grid = 3660 ; XDim_Grid = 3660 ; variables: short B01(YDim_Grid, XDim_Grid) ; B01:long_name = "Coastal_Aerosol" ; B01:_FillValue = "-1000" ; B01:scale_factor = "0.0001" ; B01:add_offset = "0.0" ; short B02(YDim_Grid, XDim_Grid) ; B02:long_name = "Blue" ; B02:_FillValue = "-1000" ; B02:scale_factor = "0.0001" ; B02:add_offset = "0.0" ; short B03(YDim_Grid, XDim_Grid) ; B03:long_name = "Green" ; B03:_FillValue = "-1000" ; B03:scale_factor = "0.0001" ; B03:add_offset = "0.0" ; short B04(YDim_Grid, XDim_Grid) ; B04:long_name = "Red" ; B04:_FillValue = "-1000" ; B04:scale_factor = "0.0001" ; B04:add_offset = "0.0" ; short B05(YDim_Grid, XDim_Grid) ; B05:long_name = "Red_Edge1" ; B05:_FillValue = "-1000" ; B05:scale_factor = "0.0001" ; B05:add_offset = "0.0" ; short B06(YDim_Grid, XDim_Grid) ; B06:long_name = "Red_Edge2" ; B06:_FillValue = "-1000" ; B06:scale_factor = "0.0001" ; B06:add_offset = "0.0" ; short B07(YDim_Grid, XDim_Grid) ; B07:long_name = "Red_Edge3" ; B07:_FillValue = "-1000" ; B07:scale_factor = "0.0001" ; B07:add_offset = "0.0" ; short B08(YDim_Grid, XDim_Grid) ; B08:long_name = "NIR_Broad" ; B08:_FillValue = "-1000" ; B08:scale_factor = "0.0001" ; B08:add_offset = "0.0" ; short B8A(YDim_Grid, XDim_Grid) ; B8A:long_name = "NIR_Narrow" ; B8A:_FillValue = "-1000" ; B8A:scale_factor = "0.0001" ; B8A:add_offset = "0.0" ; short B09(YDim_Grid, XDim_Grid) ; B09:long_name = "Water_Vapor" ; B09:_FillValue = "-1000" ; B09:scale_factor = "0.0001" ; B09:add_offset = "0.0" ; short B10(YDim_Grid, XDim_Grid) ; B10:long_name = "Cirrus" ; B10:_FillValue = "-1000" ; B10:scale_factor = "0.0001" ; B10:add_offset = "0.0" ; short B11(YDim_Grid, XDim_Grid) ; B11:long_name = "SWIR1" ; B11:_FillValue = "-1000" ; B11:scale_factor = "0.0001" ; B11:add_offset = "0.0" ; short B12(YDim_Grid, XDim_Grid) ; B12:long_name = "SWIR2" ; B12:_FillValue = "-1000" ; B12:scale_factor = "0.0001" ; B12:add_offset = "0.0" ; ubyte QA(YDim_Grid, XDim_Grid) ; QA:_FillValue = 255UB ; QA:QA\ description = "Bits are listed from the MSB (bit 7) to the LSB (bit 0): \n7-6 aerosol:\n 00 - climatology\n 01 - low\n 10 - average\n 11 - high\n5 water\n4 snow/ice\n3 cloud shadow\n2 adjacent to cloud\n1 cloud\n0 cirrus\n" ;

// global attributes: :PRODUCT_URI = "S2A_MSIL1C_20200104T174731_N0208_R098_T13SDD_20200104T211121.SAFE" ; :L1C_IMAGE_QUALITY = "NONE" ; :SPACECRAFT_NAME = "Sentinel-2A" ; :TILE_ID = "S2A_OPER_MSI_L1C_TL_MTI__20200104T211121_A023690_T13SDD_N02.08" ; :DATASTRIP_ID = "S2A_OPER_MSI_L1C_DS_MTI__20200104T211121_S20200104T175119_N02.08" ; :PROCESSING_BASELINE = "02.08" ; :SENSING_TIME = "2020-01-04T17:53:07.422642Z" ; :L1_PROCESSING_TIME = "2020-01-04T21:56:59.9799Z" ; :HORIZONTAL_CS_NAME = "WGS84 / UTM zone 13N" ; :HORIZONTAL_CS_CODE = "EPSG:32613" ; :NROWS = "3660" ; :NCOLS = "3660" ; :SPATIAL_RESOLUTION = "30" ; :ULX = 399960. ; :ULY = 4400040. ; :MEAN_SUN_ZENITH_ANGLE(B01) = 64.3317331872464 ; :MEAN_SUN_AZIMUTH_ANGLE(B01) = 161.102367370935 ; :MEAN_VIEW_ZENITH_ANGLE(B01) = 9.9000815211444 ; :MEAN_VIEW_AZIMUTH_ANGLE(B01) = 103.57344530063 ; :spatial_coverage = 46s ; :cloud_coverage = 100s ; :ACCODE = "LaSRCS2AV3.5.5" ; :arop_s2_refimg = "NONE" ; :arop_ncp = 0 ; :arop_rmse(meters) = 0. ; :arop_ave_xshift(meters) = 0. ; :arop_ave_yshift(meters) = 0. ; :HLS_PROCESSING_TIME = "2020-01-11T07:00:37Z" ; :NBAR_Solar_Zenith = 43.5498278763703 ; :AngleBand = 0UB, 1UB, 2UB, 3UB, 4UB, 5UB, 6UB, 7UB, 8UB, 9UB, 10UB, 11UB, 12UB ; :MSI\ band\ 01\ bandpass\ adjustment\ slope\ and\ offset = "0.995900, -0.000200" ; :MSI\ band\ 02\ bandpass\ adjustment\ slope\ and\ offset = "0.977800, -0.004000" ; :MSI\ band\ 03\ bandpass\ adjustment\ slope\ and\ offset = "1.005300, -0.000900" ; :MSI\ band\ 04\ bandpass\ adjustment\ slope\ and\ offset = "0.976500, 0.000900" ; :MSI\ band\ 8a\ bandpass\ adjustment\ slope\ and\ offset = "0.998300, -0.000100" ; :MSI\ band\ 11\ bandpass\ adjustment\ slope\ and\ offset = "0.998700, -0.001100" ; :MSI\ band\ 12\ bandpass\ adjustment\ slope\ and\ offset = "1.003000, -0.001200" ; :StructMetadata.0 = "GROUP=SwathStructure\nEND_GROUP=SwathStructure\nGROUP=GridStructure\n\tGROUP=GRID_1\n\t\tGridName=\"Grid\"\n\t\tXDim=3660\n\t\tYDim=3660\n\t\tPixelSize=(30.0,30.0)\n\t\tUpperLeftPointMtrs=(399960.000000,4400040.000000)\n\t\tLowerRightMtrs=(509760.000000,4290240.000000)\n\t\tProjection=GCTP_UTM\n\t\tZoneCode=13\n\t\tSphereCode=12\n\t\tDatum=WGS84\n\t\tGridOrigin=HDFE_GD_UL\n\t\tGROUP=Dimension\n\t\t\tOBJECT=Dimension_1\n\t\t\t\tDimensionName=\"YDim_Grid\"\n\t\t\t\tSize=3660\n\t\t\tEND_OBJECT=Dimension_1\n\t\t\tOBJECT=Dimension_2\n\t\t\t\tDimensionName=\"XDim_Grid\"\n\t\t\t\tSize=3660\n\t\t\tEND_OBJECT=Dimension_2\n\t\tEND_GROUP=Dimension\n\t\tGROUP=DimensionMap\n\t\tEND_GROUP=DimensionMap\n\t\tGROUP=IndexDimensionMap\n\t\tEND_GROUP=IndexDimensionMap\n\t\tGROUP=DataField\n\t\t\tOBJECT=DataField_1\n\t\t\t\tDataFieldName=\"B01\"\n\t\t\t\tDataType=DFNT_INT16\n\t\t\t\tDimList=(\"YDim_Grid\",\"XDim_Grid\")\n\t\t\tEND_OBJECT=DataField_1\n\t\t\tOBJECT=DataField_2\n\t\t\t\tDataFieldName=\"B02\"\n\t\t\t\tDataType=DFNT_INT16\n\t\t\t\tDimList=(\"YDim_Grid\",\"XDim_Grid\")\n\t\t\tEND_OBJECT=DataField_2\n\t\t\tOBJECT=DataField_3\n\t\t\t\tDataFieldName=\"B03\"\n\t\t\t\tDataType=DFNT_INT16\n\t\t\t\tDimList=(\"YDim_Grid\",\"XDim_Grid\")\n\t\t\tEND_OBJECT=DataField_3\n\t\t\tOBJECT=DataField_4\n\t\t\t\tDataFieldName=\"B04\"\n\t\t\t\tDataType=DFNT_INT16\n\t\t\t\tDimList=(\"YDim_Grid\",\"XDim_Grid\")\n\t\t\tEND_OBJECT=DataField_4\n\t\t\tOBJECT=DataField_5\n\t\t\t\tDataFieldName=\"B05\"\n\t\t\t\tDataType=DFNT_INT16\n\t\t\t\tDimList=(\"YDim_Grid\",\"XDim_Grid\")\n\t\t\tEND_OBJECT=DataField_5\n\t\t\tOBJECT=DataField_6\n\t\t\t\tDataFieldName=\"B06\"\n\t\t\t\tDataType=DFNT_INT16\n\t\t\t\tDimList=(\"YDim_Grid\",\"XDim_Grid\")\n\t\t\tEND_OBJECT=DataField_6\n\t\t\tOBJECT=DataField_7\n\t\t\t\tDataFieldName=\"B07\"\n\t\t\t\tDataType=DFNT_INT16\n\t\t\t\tDimList=(\"YDim_Grid\",\"XDim_Grid\")\n\t\t\tEND_OBJECT=DataField_7\n\t\t\tOBJECT=DataField_8\n\t\t\t\tDataFieldName=\"B08\"\n\t\t\t\tDataType=DFNT_INT16\n\t\t\t\tDimList=(\"YDim_Grid\",\"XDim_Grid\")\n\t\t\tEND_OBJECT=DataField_8\n\t\t\tOBJECT=DataField_9\n\t\t\t\tDataFieldName=\"B8A\"\n\t\t\t\tDataType=DFNT_INT16\n\t\t\t\tDimList=(\"YDim_Grid\",\"XDim_Grid\")\n\t\t\tEND_OBJECT=DataField_9\n\t\t\tOBJECT=DataField_10\n\t\t\t\tDataFieldName=\"B09\"\n\t\t\t\tDataType=DFNT_INT16\n\t\t\t\tDimList=(\"YDim_Grid\",\"XDim_Grid\")\n\t\t\tEND_OBJECT=DataField_10\n\t\t\tOBJECT=DataField_11\n\t\t\t\tDataFieldName=\"B10\"\n\t\t\t\tDataType=DFNT_INT16\n\t\t\t\tDimList=(\"YDim_Grid\",\"XDim_Grid\")\n\t\t\tEND_OBJECT=DataField_11\n\t\t\tOBJECT=DataField_12\n\t\t\t\tDataFieldName=\"B11\"\n\t\t\t\tDataType=DFNT_INT16\n\t\t\t\tDimList=(\"YDim_Grid\",\"XDim_Grid\")\n\t\t\tEND_OBJECT=DataField_12\n\t\t\tOBJECT=DataField_13\n\t\t\t\tDataFieldName=\"B12\"\n\t\t\t\tDataType=DFNT_INT16\n\t\t\t\tDimList=(\"YDim_Grid\",\"XDim_Grid\")\n\t\t\tEND_OBJECT=DataField_13\n\t\t\tOBJECT=DataField_14\n\t\t\t\tDataFieldName=\"QA\"\n\t\t\t\tDataType=DFNT_UINT8\n\t\t\t\tDimList=(\"YDim_Grid\",\"XDim_Grid\")\n\t\t\tEND_OBJECT=DataField_14\n\t\tEND_GROUP=DataField\n\t\tGROUP=MergedFields\n\t\tEND_GROUP=MergedFields\n\tEND_GROUP=GRID_1\nEND_GROUP=GridStructure\nGROUP=PointStructure\nEND_GROUP=PointStructure\nEND\n" ; } ```

Could you try xr.open_dataset(f, decode_cf=False). It could also be helpful to know the output of ncdump - k <file> and ncdump -h <file>

I have also tried a clean install in a fresh conda environment with python 3.7, and receive this traceback now upon calling xr.open_dataset(f), which previously at least was able to read the metadata. output of xr.show_versions() below that.

```

RuntimeError Traceback (most recent call last) <ipython-input-16-de78b6c36cbe> in <module> 1 # ds = xr.open_dataset(f.replace('.hdf', '.h4')) ----> 2 ds = xr.open_dataset(f) 3 ds['B01'].values

C:\software\Anaconda3\envs\x-python\lib\site-packages\xarray\backends\api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs, use_cftime, decode_timedelta) 543 544 with close_on_error(store): --> 545 ds = maybe_decode_store(store) 546 547 # Ensure source filename always stored in dataset object (GH issue #2550)

C:\software\Anaconda3\envs\x-python\lib\site-packages\xarray\backends\api.py in maybe_decode_store(store, lock) 457 drop_variables=drop_variables, 458 use_cftime=use_cftime, --> 459 decode_timedelta=decode_timedelta, 460 ) 461

C:\software\Anaconda3\envs\x-python\lib\site-packages\xarray\conventions.py in decode_cf(obj, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta) 580 encoding = obj.encoding 581 elif isinstance(obj, AbstractDataStore): --> 582 vars, attrs = obj.load() 583 extra_coords = set() 584 file_obj = obj

C:\software\Anaconda3\envs\x-python\lib\site-packages\xarray\backends\common.py in load(self) 110 """ 111 variables = FrozenDict( --> 112 (_decode_variable_name(k), v) for k, v in self.get_variables().items() 113 ) 114 attributes = FrozenDict(self.get_attrs())

C:\software\Anaconda3\envs\x-python\lib\site-packages\xarray\backends\netCDF4_.py in get_variables(self) 398 def get_variables(self): 399 dsvars = FrozenDict( --> 400 (k, self.open_store_variable(k, v)) for k, v in self.ds.variables.items() 401 ) 402 return dsvars

C:\software\Anaconda3\envs\x-python\lib\site-packages\xarray\core\utils.py in FrozenDict(args, kwargs) 440 441 def FrozenDict(args, kwargs) -> Frozen: --> 442 return Frozen(dict(*args, kwargs)) 443 444

C:\software\Anaconda3\envs\x-python\lib\site-packages\xarray\backends\netCDF4_.py in <genexpr>(.0) 398 def get_variables(self): 399 dsvars = FrozenDict( --> 400 (k, self.open_store_variable(k, v)) for k, v in self.ds.variables.items() 401 ) 402 return dsvars

C:\software\Anaconda3\envs\x-python\lib\site-packages\xarray\backends\netCDF4_.py in open_store_variable(self, name, var) 374 # netCDF4 specific encoding; save _FillValue for later 375 encoding = {} --> 376 filters = var.filters() 377 if filters is not None: 378 encoding.update(filters)

netCDF4_netCDF4.pyx in netCDF4._netCDF4.Variable.filters()

netCDF4_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()

RuntimeError: NetCDF: Attempting netcdf-4 operation on netcdf-3 file ```

xr.show_versions():

INSTALLED VERSIONS

commit: None python: 3.7.8 | packaged by conda-forge | (default, Jul 31 2020, 01:53:57) [MSC v.1916 64 bit (AMD64)] python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 158 Stepping 9, GenuineIntel byteorder: little LC_ALL: None LANG: None LOCALE: None.None libhdf5: 1.10.6 libnetcdf: 4.7.4

xarray: 0.16.1 pandas: 1.1.3 numpy: 1.19.2 scipy: 1.5.2 netCDF4: 1.5.4 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.2.1 nc_time_axis: None PseudoNetCDF: None rasterio: 1.1.7 cfgrib: None iris: None bottleneck: None dask: None distributed: None matplotlib: 3.3.2 cartopy: None seaborn: 0.11.0 numbagg: None pint: None setuptools: 49.6.0.post20201009 pip: 20.2.4 conda: None pytest: None IPython: 7.18.1 sphinx: None