issue_comments

Closing this as stale and out of date with our current backends. @swnesbitt (or others) - feel free to open a new PR if you feel there is more to do here.

The only way we could make reading a gzipped netCDF4 file is to load the entire file into memory. That's why we didn't support this before. It's also less relevant for netCDF4, because netCDF4 supports in-file compression directly.

With netCDF3, we can use scipy's netcdf reader, which supports Python file objects. But netCDF4-Python does not support Python file objects.

This issue is concerned about supporting paths ending with .gz in remote URLs, which are not local files but rather files exposed via the OpenDAP protocol over a network. If the DAP server can server a file with a .gz extension then xarray should be OK with it, too.

@tsaoyu - I don't think anyone has worked on developing a test case for this feature. I assume @swnesbitt would appreciate help there.

Currently the way we handle this is that the only test that accessing remote resources is the pydap test, which only runs in one build (not required to pass).

I am reluctant to merge this without having any way to test the logic. Without automated tests this issue is likely to recur.

That said, I suppose we could leave the refactoring for a TODO. Let's add a note on that and also one minimal test to verify that we raise an error if you try to use engine='netcdf4' with a path to local gzipped file.

@shoyer - are we okay with the final logic here?

@swnesbitt - can we get an entry in the what's new docs?

Regarding testing, it would be good to have a test module that includes grabbing files for local access and accessing openDAP datasets (from UNIDATA?) to ensure that the installed backends are all working as expected.

I'd be a little hesitant to enforce that we get access to opendap (or other remote datasets) on travis. I've seen this come back to bite us in the past. We can allow tests to fail on travis and just print a warning via pytest if it is something we really want to see tested in that way.

We do have existing tests for backends: https://github.com/pydata/xarray/blob/master/xarray/test/test_backends.py

This includes a test accessing an OpenDAP dataset from the OpenDAP test server (via pydap, at the end). But in my experience, there servers are somewhat unreliable (maybe available 90%), so we don't require that test to pass for the build to pass. Also, even in the best case scenario network access is slow. So it would be nice to modularize this enough that our logic is testable without actually using opendap.

I think this is probably correct, but the heuristics here are starting to get convoluted enough that I worry about test coverage. Is there any way we can test this? Maybe try to pull the gzip logic into a helper function (an extended variant of _get_default_engine) that we could test?