This works with latest libraries.

@rabernat - do think this was resolved? If I'm understanding the thread correctly, it seems this was a libnetcdf version issue. Feel free to reopen if I've got that wrong.

h10edf3e_1 contains the timeout fix and is build against hdf5 1.10.2. The conda-forge h9cd6fdc_11 build is against hdf5 1.10.3 perhaps that makes a different?

There are many variables at play here. The env that solved it in https://github.com/pydata/xarray/issues/2503#issuecomment-432645477 seems quite different from the env where the problem happened, including an xarray dev version. I'm not sure hdf5 is a good candidate to blame :smile:

h10edf3e_1 contains the timeout fix and is build against hdf5 1.10.2. The conda-forge h9cd6fdc_11 build is against hdf5 1.10.3 perhaps that makes a different?

In defaults libnetcdf4 4.6.1 build 1 and above contain the timeout fix, build 0 has the original timeout.

Thanks @jjhelmus! I guess that info and https://github.com/pydata/xarray/issues/2503#issuecomment-432483817 eliminates the timeout issue from the equation.

In defaults libnetcdf4 4.6.1 build 1 and above contain the timeout fix, build 0 has the original timeout.

Oh, I didn't even catch that the original was on defaults.

That version has the fix for the issue.

I know that @jjhelmus ported the fix to defaults but I'm not sure which build number has it, and/or if the previous one was remove, b/c defaults builds are not as transparent as conda-forge's :smile:

He can probably say more about that.

That version has the fix for the issue.

The version that is working in @rabernat's esgf binder env is: libnetcdf 4.6.1 h9cd6fdc_11 conda-forge

The original version of libnetcdf in @rabernat 's environment definitely had the opendap timeout issue. Not sure if that's the root cause of the problem, or not, but it's suspect.

I fired up my notebook on @rabernat's binder env and it worked fine also: https://nbviewer.jupyter.org/gist/rsignell-usgs/aebdac44a1d773b99673cb132c2ef5eb

I created a binderized version of this issue with the latest dev xarray and fresh installs of all other packages: https://github.com/rabernat/pangeo_esgf_demo

It appears to work fine!

``` $ conda list libnetcdf

packages in environment at /opt/conda:

Name Version Build Channel

libnetcdf 4.6.1 h10edf3e_1 defaults ```

@lesserwhirls That's an interesting idea. (@rsignell-usgs That's the one.)

@rabernat What version of the conda-forge libnetcdf package is deployed wherever you're running?

@rabernat have you tried using the development version of xarray? I think we fixed a few serialization/ netCDF4 bugs with the backends refactor.

@lesserwhirls , is this the issue you are referring to? https://github.com/Unidata/netcdf4-python/issues/836

@jhamman, doesn't this dask status plot tell us that multiple workers are connecting and getting data?

So for just a short time period (in this case 50 time steps, 2 chunks in time), it would usually work:

The "short time period" makes me wonder...@dopplershift - could this be due to the netCDF-C / curl timeout issue you mentioned today?

@rsignell-usgs - are you able to tell if multiple processes (workers) have authenticated on the server side? I think this detail would really help us isolate the problem.

Perhaps it's also worth mentioning that I don't see any errors on the THREDDS server side on either the tomcat catalina or thredds threddsServlet logs. @lesserwhirls, any ideas?

FWIW, in my workflow there was nothing fundamentally wrong, meaning that the requests worked for a while, but eventually would die with the NetCDF: Malformed or inaccessible DAP DDS message.

So for just a short time period (in this case 50 time steps, 2 chunks in time), it would usually work: https://nbviewer.jupyter.org/gist/rsignell-usgs/1155c76ed3440858ced8132e4cd81df4

I tried a similar workflow last week with an AWS kubernetes cluster with opendap endpoints and it also failed: https://nbviewer.jupyter.org/gist/rsignell-usgs/8583ea8f8b5e1c926b0409bd536095a9

I thought it was likely some intermittent problem that wasn't handled well. In my case after a while I get: distributed.worker - WARNING - Compute Failed Function: getter args: (ImplicitToExplicitIndexingAdapter(array=CopyOnWriteArray(array=LazilyOuterIndexedArray(array=_ElementwiseFunctionArray(LazilyOuterIndexedArray(array=<xarray.backends.netCDF4_.NetCDF4ArrayWrapper object at 0x7ff93cbbd828>, key=BasicIndexer((slice(None, None, None), slice(None, None, None), slice(None, None, None), slice(None, None, None)))), func=functools.partial(<function _apply_mask at 0x7ff945421378>, encoded_fill_values={1e+37}, decoded_fill_value=nan, dtype=dtype('float64')), dtype=dtype('float64')), key=BasicIndexer((slice(None, None, None), slice(None, None, None), slice(None, None, None), slice(None, None, None)))))), (slice(375, 400, None), slice(0, 7, None), slice(0, 670, None), slice(0, 300, None))) kwargs: {} Exception: OSError(-72, 'NetCDF: Malformed or inaccessible DAP DDS')

Just so I'm clear on how the workflow looks: 1. Open dataset with NetCDF/OPeNDAP 2. Serialize NetCDFDataStore (pickle? netcdf file?) 3. Ship to Dask workers 4. Reconstitute NetCDFDataStore

Certainly does seem like there's something stale in what the remote workers are getting. Confused why it works for the others, though.

I can prioritize this a bit and dig in to see what I can figure out--though I'm teaching through tomorrow. May be able to dig into this while at ECMWF.

This is fairly high priority for me, as it relates to the ongoing project to access CMIP6 data from an ESGF node running in google cloud (https://github.com/pangeo-data/pangeo/issues/420).

I'm wondering if there is some authentication that is not being properly distributed to the workers. I'm not actually sure how opendap works in this case. Perhaps @dopplershift has some ideas here? Has anyone, maybe @rsignell-usgs, used the kubernetes cluster with opendap endpoints?