Both the warning message and the upstream anaconda issue seem like good ideas to me.

If upgrating to a newer version of netcdf4-python isn't an option we might need to figure out a workaround for xarray....

It seems that anaconda is still distributing netCDF4 1.2.4, which doesn't help here.

Can you reproduce this just using netCDF4-python?

Try: ``` import netCDF4 ds = netCDF4.Dataset(path)

print(ds)

print(ds.filepath()) ```

If so, it would be good to file a bug upstream.

Actually, it looks like this might be https://github.com/Unidata/netcdf4-python/issues/506

I think there is probably a bug buried inside the netCDF4.Dataset.filepath() method somewhere. For example, on netCDF4-python 1.2.4, this would crash if you have any non-ASCII characters in the path. But that doesn't seem to be the issue here.

@braaannigan can you try adding print(repr(path)) to is_remote_uri() so we can see exactly what these offending strings look like?

re.match(pattern, string) is equivalent to re.search('^' + pattern, string), so arguably this is a cleaner solution anyways. But ideally I'd like to understand why this is a problem for you, so we can fix the underlying cause and not do it again.

@braaannigan what about replacing re.search('^https?\://', path) with re.match('https?\://', path)? Can you share the output of running python -c 'import sys; print(sys.getfilesystemencoding())' at the command line? Also, please try engine='scipy' or engine='h5netcdf' with open_dataset. The output of xarray.show_versions() would also be helpful.

@braaannigan Can you share the name of your problematic file?

One possibility is that re.search() is not thread-safe, even though I don't think we call is_remote_uri from multiple threads. We can test that by adding a lock, and seeing if that resolves the issue. Try replacing is_remote_uri with: ```python import threading

LOCK = threading.Lock()

def is_remote_uri(path): with LOCK: return bool(re.search('^https?\://', path)) ```

Only if the coords are tridimensional..

(405*282*37)*20*8 bytes = 676 MB, so running out of memory here seems plausible to me.

It sounds weird. Even if all the 20 variables he's dropping were coords on the longest dim, and the code was loading them up into memory and then dropping them (that would be wrong - but I didn't check the code yet to verify if that's the case), then we're talking about... 405*20*73=~690k points? That's about 5mb of RAM if they're float64?

@njweber2 how large are these files? Is it feasible to upload them somewhere? If not, could you write a script that generates equivalent dummy data and reproduce the problem with that?

I think this was introduced by https://github.com/pydata/xarray/pull/1551, where we started loading coordinates that are compared for equality into memory. This speeds up open_mfdataset, but does increase memory usage.

We might consider adding an option for reduced memory usage at the price of speed. @crusaderky @jhamman @rabernat any thoughts?