I'm not really sure what to think any more — we have had a real, consistent issue that seemed to fit the description of this issue which went away with one of the fixes above (using single threading), but using local files at the moment seems to remove the error even with the current version of xarray and either parallel option.

Ok I downloaded the two files and indeed there is no error with parallel=True nor parallel=False.

@jhamman Yes, using the PR version of xarray, with parallel=True I met the error but with parallel=False I did not.

I was able to reproduce the error with the current version of xarray and then have it work with the new version. Here is what I did:

Make new environment conda create -n test_xarray xarray netcdf4 dask

Check version ``` (test_xarray) kthyng@adams ~ % conda list xarray

packages in environment at /Users/kthyng/miniconda3/envs/test_xarray:

Name Version Build Channel

xarray 2023.3.0 pyhd8ed1ab_0 conda-forge ```

In python: import xarray as xr urls = ["https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/WCOFS/MODELS/2023/03/31/nos.wcofs.2ds.n001.20230331.t03z.nc", "https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/WCOFS/MODELS/2023/03/31/nos.wcofs.2ds.n002.20230331.t03z.nc"] xr.open_mfdataset(urls) returns the following the first time xr.open_mfdataset(urls) is run but the second time it runs fine. OSError: [Errno -70] NetCDF: DAP server error: 'https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/WCOFS/MODELS/2023/03/31/nos.wcofs.2ds.n002.20230331.t03z.nc'

Next I used the PR version of xarray and reran the code above and then it was able to read in ok on the first try.

Note: after a week or so those files won't work and will have to be updated with something more current but the pattern to use is clear from the file names.

@jhamman Sorry for my delay — I started this the other day and got waylaid. I'll try to get back to it today or tomorrow.

@ocefpaf and all: thank you! What a mysterious error this has been. Using the workaround

import dask dask.config.set(scheduler="single-threaded")

did indeed avoid the issue for me.

@benbovy Ah, I see you mean under "Relax all constraints related to “dimension (index) coordinates” in Xarray". Ok, thank you for clarifying that for me! (I wasn't sure what the second item meant in the list of lists.)

I've looked through the github issues associated with the explicit indices, but can't quite tell if I can use them to load FVCOM model output. In any case I just updated and tried without doing anything new and it didn't work:

import xarray as xr url = 'https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/SFBOFS/MODELS/2022/06/30/nos.sfbofs.fields.f014.20220630.t09z.nc' # this file will not be available in a few days but one for the present day will be available ds = xr.open_dataset(url, drop_variables='Itime2')

Same error message as before:

MissingDimensionsError: 'siglay' has more than 1-dimension and the same name as one of its dimensions ('siglay', 'node'). xarray disallows such variables because they conflict with the coordinates used to label dimensions.

@spencerkclark Interesting! Good I am glad it is fixed with the dev version. Great if this work up can be used for a test.

Mostly, I wanted to also get this documented since it took forever to track down the issue — maybe it will save someone else some time when they are googling.

What?! Whoa I did not know about the preprocess option and it looks really powerful! I have been getting the derived datasets to work but I think this would do the job in a more simple and easy-to-understand way. I will give it a try.

intake-xarray should now work with open_mfdataset — I added this as an option, though it's probably not in a release yet.

Thanks @d70-t for the idea! I haven't tried out the derived datasets capabilities in intake, but I'll give them a try. Sounds like they could be pretty powerful.

Huh! Weird! I had tried accessing a particular value of u with netCDF as a test and it had worked fine and hadn't worried about it after that. I just now tried ocean_time and it works for particular indices (like d['ocean_time'][0]), but as you said doesn't work if I put in d['ocean_time'][:].

Thanks for everyone's help, I'll work on the thredds end of things.