issue_comments

Thanks! @aaron-kaplan

Thanks to the holiday lull, I finally found some time to finish implementing URL-decoding for querystrings in the IRI Data Library server software. The example now works for me with xarray 2022.12.0, netcdf4 1.6.2, and libnetcdf 4.8.1.

Thank you for the issue discussion and temporary solution from @scottyhq! I was able to use the temporary solution to get the data I needed from IRI at the moment. The example still generates error while I tried and have some similar error when downloading the NMME hindcasts. Hope the IRI software issue will be clear soon!

The OPeNDAP server at iridl.ldeo.columbia.edu doesn't accept URL-encoded querystrings. It never did, but that used to be ok because older versions of netcdf-c didn't encode the querystrings. More recent versions do. So a change in netcdf-c is the proximal cause of the example breaking, but the root of the problem is on the server side.

We (IRI) are working on fixing the server software.

The message:

"The identifier `tmax.tmax%5b0%5d%5b0:3:620%5d%5b0:3:1404%5d' is not in the dataset."

makes me wonder if there's some issue with how the webserver is handling the escaping of certain characters (like e.g. [)

Hi, did anyone in the meantime gain insights on the cause of this issue? @scottyhq @dopplershift @rabernat ?

@scottyhq Sorry yes that was indeed the problem, was installing packages with pip and that caused incompatibility, thanks for looking into it. Would be great nevertheless to figure out the root of the issue.

@Tinkaa I looked into this a bit more and I suspect the way you are installing packages (conda, pip?) is important to bring in compatible libnetcdf behind the scenes (You're xr.show_versions() shows new libnetcdf: 4.7.4 but older netCDF4: 1.5.0.1 making me think you installed and older version netcdf into an existing environment without any dependency resolution).

This works for me on mac0S: conda create -c conda-forge -n opendap_working python xarray netcdf4==1.5.1 matplotlib Note that I found the explicit pin important. Perhaps because it brings in libnetcdf=4.6.2 behind the scenes? Here is my full list of working versions:

``` INSTALLED VERSIONS

commit: None python: 3.7.10 | packaged by conda-forge | (default, Feb 19 2021, 15:59:12) [Clang 11.0.1 ] python-bits: 64 OS: Darwin OS-release: 20.3.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.4 libnetcdf: 4.6.2

xarray: 0.16.2 pandas: 1.2.2 numpy: 1.20.1 scipy: None netCDF4: 1.5.0.1 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.4.1 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: None distributed: None matplotlib: 3.3.4 cartopy: None seaborn: None numbagg: None pint: None setuptools: 49.6.0.post20210108 pip: 21.0.1 conda: None pytest: None IPython: None sphinx: None ```

Hopefully some folks that know more about opendap could chime in (@dopplershift, @rabernat ?) to help get to the root of this issue. For what it's worth, I noticed this additional log output that wasn't surfaced in the jupyter notebook in addition to the original traceback i posted: oc_open: server error retrieving url: code=3 message="The identifier `tmax.tmax%5b0%5d%5b0:3:620%5d%5b0:3:1404%5d' is not in the dataset."oc_open: server error retrieving url: code=3 message="The identifier `tmax.tmax%5b0%5d%5b0:3:620%5d%5b0:3:1404%5d' is not in the dataset."oc_open: server error retrieving url: code=3 message="The identifier `tmax.tmax%5b0%5d%5b0:3:620%5d%5b0:3:1404%5d' is not in the dataset."oc_open: server error retrieving url: code=3 message="The identifier `tmax.tmax%5b0%5d%5b0:3:620%5d%5b0:3:1404%5d' is not in the dataset." oc_open: server error retrieving url: code=3 message="The identifier `tmax.tmax%5b0%5d%5b0:3:620%5d%5b0:3:1404%5d' is not in the dataset."^CTraceback (most recent call last): File "/Users/scott/miniconda3/envs/opendap_test/lib/python3.9/site-packages/xarray/backends/common.py", line 52, in robust_getitem return array[key] File "src/netCDF4/_netCDF4.pyx", line 4420, in netCDF4._netCDF4.Variable.__getitem__ File "src/netCDF4/_netCDF4.pyx", line 5363, in netCDF4._netCDF4.Variable._get File "src/netCDF4/_netCDF4.pyx", line 1950, in netCDF4._netCDF4._ensure_nc_success RuntimeError: NetCDF: file not found

Just confirming I have the same error/traceback with the example you gave. But this error persists when pinning netCDF4 to version 1.5.1.. Got to the issue because I got the same error when trying to clip another IRI dataset with OpenDAP.

Environment:

$ INSTALLED VERSIONS
------------------
commit: None
python: 3.8.5 (default, Nov 18 2020, 15:31:37) 
[Clang 12.0.0 (clang-1200.0.31.1)]
python-bits: 64
OS: Darwin
OS-release: 20.3.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.UTF-8
libhdf5: 1.12.0
libnetcdf: 4.7.4

xarray: 0.16.2
pandas: 1.1.3
numpy: 1.19.1
scipy: 1.5.4
netCDF4: 1.5.0.1
pydap: installed
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.3.0
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.1.7
cfgrib: None
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: 3.3.1
cartopy: 0.18.0
seaborn: 0.11.0
numbagg: None
pint: None
setuptools: 50.3.2
pip: 20.3.3
conda: None
pytest: None
IPython: 7.19.0
sphinx: None