issue_comments

Thank you, @dcherian

There are 3 approvals and @martindurant has been quite patient! Thanks. merging. We can update the min_deps_check later.

if you wait two days the minimum version policy check should pass with aiobotocore=1.1.2, too.

Can someone please explain the minimum version policy that is failing ``` Package Required Policy Status

aiobotocore 1.1.2 (2020-08-18) 0.12 (2020-02-24) > (!) (w) ```

could you add aiobotocore to environment.yml, too? py38-all-but-dask should be environment.yml without dask / distributed. Also, could you merge in recent master to resolve the conflicts?

@raybellwaves , might I paraphrase to "this PR is useful, please merge!" ?

I think my Q on SO is related to this PR https://stackoverflow.com/questions/66145459/open-mfdataset-on-remote-zarr-store-giving-zarr-errors-groupnotfounderror

Was looking at reading a remote zarr store using open_mfdataset

@martindurant suggested putting the single "file" (mapping) in a list which works ds = xr.open_mfdataset([file], engine="zarr")

but I also wanted to test the other suggestion

ds = xr.open_mfdataset(uri, engine="zarr", backend_kwargs=dict(storage_options={'anon': True}))

On current xarray master I get Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/ray.bell/Documents/PYTHON_dev/xarray/xarray/backends/api.py", line 884, in open_mfdataset raise OSError("no files to open") OSError: no files to open

On this branch it works ``` git clone https://github.com/martindurant/xarray.git git checkout fsspec_mk2 conda create -c conda-forge -n xarray-tests python=3.8 conda env update -f ci/requirements/environment.yml conda activate xarray-tests pip install -e . pip install s3fs

import xarray as xr uri = "s3://era5-pds/zarr/2020/12/data/eastward_wind_at_10_metres.zarr" ds = xr.open_mfdataset(uri, engine="zarr", backend_kwargs=dict(storage_options={'anon': True})) ```

I have decided, on reflection, to back away on the scope here and only implement for zarr for now, since, frankly, I am confused about what should happen for other backends, and they are not tested. Yes, some of them are happy to accept file-like objects, but others either don't do that at all, or want the URL passing through. My code would have changed how things were handled, depending on whether it passed through open_dataset or open_mfdataset. Best would be to set up a set of expectations as tests.

Thanks, @kmuehlbauer

@martindurant Should be fixed by #4845. Probably just needs rebase.

The RTD failure appears to be: WARNING: failed to reach any of the inventories with the following issues: intersphinx inventory 'https://scitools.org.uk/iris/docs/latest/objects.inv' not fetchable due to <class 'requests.exceptions.HTTPError'>: 404 Client Error: Not Found for url: https://scitools.org.uk/iris/docs/latest/objects.inv which, I'm afraid, doesn't mean much to me.

I am marking this PR as ready, but please ask me for specific test cases that might be relevant and should be included.

(please definitely do not merge until I've added documentation)

@martindurant with respect to the backend API (old and new) looks good to me. I'm don't know fsspec so I can't comment on that.

@martindurant it is OK to let every backend raise errors for unsupported input. So no need to add any additional logic here IMO.

Next open question: aside from zarr, few of the other backends will know what to do with fsspec's dict-like mappers. Should we prevent them from passing through? Should we attempt to distinguish between directories and files, and make fsspec file-like objects? We could just allow the backends to fail later on incorrect input.

Looking over the changes to is_remote_uri(), it looks like this should resolve the issue I had at #4691!

pint errors in xarray/tests/test_units.py::TestVariable appear unrelated

Question: should HTTP URLs be passed through unprocessed as before? I think that might be required by some of the netCDF engines, but we probably don't test this.

Docs to be added