Hi, check out the work I did on the RTD requirements file here, it may help: https://github.com/pydata/xarray/pull/3358

looking at this once again, there are a lot of projects that don't install (eg aiohttp (build), but I'm not sure how this is set up or which parts of their configuration comes from somewhere other than their .readthedocs.yml).

Also, it seems we are using the old version of readthedocs.yml.

And finally, it seems strange to me that the scipy19-docs fails but master doesn't, which means that master does something different?

As I don't have access to readthedocs I can't really experiment on these suggestions (making this informed guessing at best), so I would like to leave the tracking down and fixing to someone else.

I'm currently not planning to because I will probably be offline for the next few days

Edit: the only workaround I can imagine is building the docs without installing (which is mentioned in the docs), but I neither know why the current configuration is the way it is or how building the docs without installing works with RTD

Is someone planning to open this RTD issue? Or is there another workaround? I would love to get this scipy19-docs branch merged to master, but this seems to be standing in the way.

Yikes, OK. Yes, it would be good to open an issue on the RTD GitHub page to discuss our options. --upgrade-strategy eager seems like a bad idea for us -- it really undermines the value of dependency pinning.

On Tue, Sep 24, 2019 at 1:31 PM keewis notifications@github.com wrote:

it seems all of these problems boil down to that single command. What gets executed is (essentially)

$ python -m pip install --upgrade --upgrade-strategy eager .

which upgrades pandas to 0.25.1, but leaves dask as is because it is an optional dependency. This, however, defeats the purpose of version pinning in the conda file (or a requirements file).

I can't find an option to tell readthedocs to install without the eager upgrade (normal upgrade is fine). Maybe ask support? The addition of the upgrade strategy happened in readthedocs/readthedocs.org#5635 https://github.com/readthedocs/readthedocs.org/pull/5635

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/pull/3199?email_source=notifications&email_token=AAJJFVULDVAUPID5WWAZAF3QLJ2JXA5CNFSM4IKVN6A2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7PWUAY#issuecomment-534735363, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJJFVRI6S3TML3E27VEYNTQLJ2JXANCNFSM4IKVN6AQ .

it seems all of these problems boil down to that single command. What gets executed is (essentially) bash $ python -m pip install --upgrade --upgrade-strategy eager . which upgrades pandas to 0.25.1, but leaves dask as is because it is an optional dependency. This, however, defeats the purpose of version pinning in the conda file (or a requirements file).

I can't find an option to tell readthedocs to install without the eager upgrade (normal upgrade is fine). Maybe ask support? The addition of the upgrade strategy happened in readthedocs/readthedocs.org#5635

OK that almost worked. Now it looks like we need to bump pandas? or dask?

``` AttributeError Traceback (most recent call last) <ipython-input-8-708052ae49bf> in <module> ----> 1 df = ds.to_dask_dataframe()

~/checkouts/readthedocs.org/user_builds/xray/conda/scipy19-docs/lib/python3.7/site-packages/xarray/core/dataset.py in to_dask_dataframe(self, dim_order, set_index) 4278 series_list.append(series) 4279 -> 4280 df = dd.concat(series_list, axis=1) 4281 4282 if set_index:

~/checkouts/readthedocs.org/user_builds/xray/conda/scipy19-docs/lib/python3.7/site-packages/dask/dataframe/multi.py in concat(dfs, axis, join, interleave_partitions) 598 axis = DataFrame._validate_axis(axis) 599 dasks = [df for df in dfs if isinstance(df, _Frame)] --> 600 dfs = _maybe_from_pandas(dfs) 601 602 if axis == 1:

~/checkouts/readthedocs.org/user_builds/xray/conda/scipy19-docs/lib/python3.7/site-packages/dask/dataframe/core.py in _maybe_from_pandas(dfs) 3461 dfs = [from_pandas(df, 1) 3462 if (is_series_like(df) or is_dataframe_like(df)) and not is_dask_collection(df) -> 3463 else df for df in dfs] 3464 return dfs 3465

~/checkouts/readthedocs.org/user_builds/xray/conda/scipy19-docs/lib/python3.7/site-packages/dask/dataframe/core.py in <listcomp>(.0) 3461 dfs = [from_pandas(df, 1) 3462 if (is_series_like(df) or is_dataframe_like(df)) and not is_dask_collection(df) -> 3463 else df for df in dfs] 3464 return dfs 3465

~/checkouts/readthedocs.org/user_builds/xray/conda/scipy19-docs/lib/python3.7/site-packages/dask/dataframe/utils.py in is_series_like(s) 503 def is_series_like(s): 504 """ Looks like a Pandas Series """ --> 505 return set(dir(s)) > {'name', 'dtype', 'groupby', 'head'} 506 507

~/checkouts/readthedocs.org/user_builds/xray/conda/scipy19-docs/lib/python3.7/site-packages/dask/dataframe/core.py in dir(self) 2554 o = set(dir(type(self))) 2555 o.update(self.dict) -> 2556 o.update(c for c in self.columns if 2557 (isinstance(c, pd.compat.string_types) and 2558 pd.compat.isidentifier(c)))

~/checkouts/readthedocs.org/user_builds/xray/conda/scipy19-docs/lib/python3.7/site-packages/dask/dataframe/core.py in <genexpr>(.0) 2555 o.update(self.dict) 2556 o.update(c for c in self.columns if -> 2557 (isinstance(c, pd.compat.string_types) and 2558 pd.compat.isidentifier(c))) 2559 return list(o)

AttributeError: module 'pandas.compat' has no attribute 'string_types' <<<------------------------------------------------------------------------- ```

:+1:

Let's see if it works: https://readthedocs.org/projects/xray/builds/9705929/

After some experimentation, it seems this is only the case if setup.py install is used, not with pip install: for me, at least, in the commands from https://github.com/pydata/xarray/pull/3199#issuecomment-534342306, replacing bash $ python setup.py install --force with bash $ python -m pip install . makes this work, even without the merge commit.

The docs on the settings mention pip_install, maybe use that?

The problem is still the same: while the output of the environment printed by sphinx lists the new version as xarray 0.13.0+22.gaeb15b56 pypi_0 pypi which in theory should be used, but building the docs prints this: xarray: 0.13.0, /home/docs/checkouts/readthedocs.org/user_builds/xray/conda/scipy19-docs/lib/python3.7/site-packages/xarray/__init__.py which does not have the version of open_dataset that allows non-nc files. Honestly, I'm no expert on imports and version management, so I don't know what the best fix for this is.

Something that comes to mind is waiting for a new release with the new version of open_dataset, but that is not an immediate fix (and only a workaround at best).

still fails with 404: https://readthedocs.org/projects/xray/builds/9704981/

Thanks @keewis . I just merged master and pushed. hopefully this works

err, well, this is somewhat confusing: I'm not sure what pulls it, but it seems something depends on xarray (installed in the creation of the environment). The version that is conda-installed is 0.13.0, so as master does not have the new version of open_dataset that does not enforce ".nc" files, the 404 happens. In short, the version from setup.py install gets shadowed, I guess because the version is 0.12.3? In that case, merging master into scipy19-docs might fix this?

Unfortunately, I don't think we can remove the second conda install. RTD insists on it to ensure that it has the build environment it needs. We could certainly ask on their GitHub support page, though.

On Mon, Sep 23, 2019 at 5:59 PM keewis notifications@github.com wrote:

(roughly) following the instructions in the build log, I can reproduce it raising a 404:

$ conda env create --quiet --name scipy19-docs --file doc/environment.yml $ conda install --yes --quiet --name scipy19-docs mock pillow sphinx sphinx_rtd_theme (scipy19-docs)$ python -m pip install -U recommonmark readthedocs-sphinx-ext (scipy19-docs)$ python setup.py install --force (scipy19-docs)$ cd doc; make html

I didn't check yet, but maybe the second conda install should not happen, as all those get installed on env creation?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/pull/3199?email_source=notifications&email_token=AAJJFVQVW3X5GAUBKXSBT7DQLFQ7LA5CNFSM4IKVN6A2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7MWVIQ#issuecomment-534342306, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJJFVVKECGIALOX4DMWUSLQLFQ7LANCNFSM4IKVN6AQ .

(roughly) following the instructions in the build log, I can reproduce it raising a 404: bash $ conda env create --quiet --name scipy19-docs --file doc/environment.yml $ conda install --yes --quiet --name scipy19-docs mock pillow sphinx sphinx_rtd_theme (scipy19-docs)$ python -m pip install -U recommonmark readthedocs-sphinx-ext (scipy19-docs)$ python setup.py install --force (scipy19-docs)$ cd doc; make html I didn't check yet, but maybe the second conda install should not happen, as all those get installed on env creation?

Unfortunately the RTD build failed!

https://readthedocs.org/projects/xray/builds/9700942/

😭😭😭

This is really trying my patience!

We got it! Thanks everyone!

Thanks @keewis

:man_facepalming: finally!

that seems to have worked, the failures should be unrelated.

I removed the newline upstream.

unless something else is wrong, either modifying the md5 file (remove the newline) or adding strip() to f.read() in tutorial.open_dataset should fix the md5 check.

Still failing. Instead of

ds = xr.tutorial.load_dataset('data/era5-2mt-2019-03-uk.grib', engine='cfgrib')

it should be

ds = xr.tutorial.load_dataset('era5-2mt-2019-03-uk.grib', engine='cfgrib')

Codecov Report

Merging #3199 into scipy19-docs will decrease coverage by 0.35%. The diff coverage is n/a.

```diff @@ Coverage Diff @@

scipy19-docs #3199 +/-

================================================ - Coverage 95.76% 95.41% -0.36%
================================================ Files 64 64
Lines 13188 13188
================================================ - Hits 12630 12583 -47
- Misses 558 605 +47 ```

Ah, oops. Updated to use that method now.

Oops I think you lost ryan's commit with xr.tutorial.load_dataset

Rebased on scipy19-docs. Hope that helps!

@zbruick Can you merge in scipy19-docs? That might fix it.

I'm not sure if this helps you, but I noticed two issues in xr.tutorial.open_dataset: first, it requires the file extension to be ".nc" (resulting in a 404 when fetching the file. The version from the scipy19-docs branch does not have that issue), and second, there is a newline in the md5 file. Modifying remotemd5 = f.read() to remotemd5 = f.read().strip() should do the trick.

I am really stumped here.

When I run python xr.tutorial.load_dataset('era5-2mt-2019-03-uk.grib', engine='cfgrib')

on the scipy19-docs branch, it works fine.

Can someone with the proper credentials (e.g. @shoyer) restart the azure build? I can't reproduce the md5 error locally.

Thanks for pushing that commit and updating the data import call. I forgot I had to switch that from my local file. Let me know if this issue is solvable on my side of things, but that seems like that's an issue with the data file?

Now we are getting MD5 checksum does not match, try downloading dataset again. Doesn't make sense to me.

https://dev.azure.com/xarray/xarray/_build/results?buildId=780

I just fixed a file naming issue in https://github.com/pydata/xarray-data/pull/18 and can confirm that my code above works for me.

It looks like the doc build is failing with this error

``` Sphinx parallel build error: nbsphinx.NotebookError: CellExecutionError in examples/ERA5-GRIB-example.ipynb:

ds = xr.load_dataset('data/era5-2mt-2019-03-uk.grib', engine='cfgrib')

---------------------------------------------------------------------------‌ FileNotFoundError‌ Traceback (most recent call last)‌ <ipython-input-2-90ecf53ec89c>‌ in ‌<module>‌ ----> 1‌ ‌ds‌ ‌=‌ ‌xr‌.‌load_dataset‌(‌'data/era5-2mt-2019-03-uk.grib'‌,‌ ‌engine‌=‌'cfgrib'‌)‌

~/work/1/s/xarray/backends/api.py‌ in ‌load_dataset‌(filename_or_obj, kwargs)‌ 256‌ ‌raise‌ ‌TypeError‌(‌"cache has no effect in this context"‌)‌ 257‌ ‌ --> 258‌ ‌with‌ ‌open_dataset‌(‌filename_or_obj‌,‌ ‌‌kwargs‌)‌ ‌as‌ ‌ds‌:‌ 259‌ ‌return‌ ‌ds‌.‌load‌(‌)‌ 260‌ ‌

~/work/1/s/xarray/backends/api.py‌ in ‌open_dataset‌(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs, use_cftime)‌ 516‌ ‌elif‌ ‌engine‌ ‌==‌ ‌"cfgrib"‌:‌ 517‌ store = backends.CfGribDataStore(‌ --> 518‌ ‌filename_or_obj‌,‌ ‌lock‌=‌lock‌,‌ ‌**‌backend_kwargs‌ 519‌ )‌ 520‌ ‌

~/work/1/s/xarray/backends/cfgrib_.py‌ in ‌init‌(self, filename, lock, backend_kwargs)‌ 41‌ ‌lock‌ ‌=‌ ‌ECCODES_LOCK‌ 42‌ ‌self‌.‌lock‌ ‌=‌ ‌ensure_lock‌(‌lock‌)‌ ---> 43‌ ‌self‌.‌ds‌ ‌=‌ ‌cfgrib‌.‌open_file‌(‌filename‌,‌ ‌‌backend_kwargs‌)‌ 44‌ ‌ 45‌ ‌def‌ ‌open_store_variable‌(‌self‌,‌ ‌name‌,‌ ‌var‌)‌:‌

/usr/share/miniconda/envs/xarray-tests/lib/python3.7/site-packages/cfgrib/dataset.py‌ in ‌open_file‌(path, grib_errors, indexpath, filter_by_keys, kwargs)‌ 601‌ ):‌ 602‌ ‌"""Open a GRIB file as a cfgrib.Dataset."""‌ --> 603‌ ‌index‌ ‌=‌ ‌open_fileindex‌(‌path‌,‌ ‌grib_errors‌,‌ ‌indexpath‌,‌ ‌filter_by_keys‌)‌ 604‌ ‌return‌ ‌Dataset‌(‌*‌build_dataset_components‌(‌index‌,‌ ‌‌kwargs‌)‌)‌

/usr/share/miniconda/envs/xarray-tests/lib/python3.7/site-packages/cfgrib/dataset.py‌ in ‌open_fileindex‌(path, grib_errors, indexpath, filter_by_keys)‌ 594‌ ‌filter_by_keys‌ ‌=‌ ‌dict‌(‌filter_by_keys‌)‌ 595‌ ‌stream‌ ‌=‌ ‌messages‌.‌FileStream‌(‌path‌,‌ ‌message_class‌=‌cfmessage‌.‌CfMessage‌,‌ ‌errors‌=‌grib_errors‌)‌ --> 596‌ ‌return‌ ‌stream‌.‌index‌(‌ALL_KEYS‌,‌ ‌indexpath‌=‌indexpath‌)‌.‌subindex‌(‌filter_by_keys‌)‌ 597‌ ‌ 598‌ ‌

/usr/share/miniconda/envs/xarray-tests/lib/python3.7/site-packages/cfgrib/messages.py‌ in ‌index‌(self, index_keys, indexpath)‌ 235‌ ‌def‌ ‌index‌(‌self‌,‌ ‌index_keys‌,‌ ‌indexpath‌=‌'{path}.{short_hash}.idx'‌)‌:‌ 236‌ ‌# type: (T.List[str], str) -> FileIndex‌ --> 237‌ ‌return‌ ‌FileIndex‌.‌from_indexpath_or_filestream‌(‌self‌,‌ ‌index_keys‌,‌ ‌indexpath‌)‌ 238‌ ‌ 239‌ ‌

/usr/share/miniconda/envs/xarray-tests/lib/python3.7/site-packages/cfgrib/messages.py‌ in ‌from_indexpath_or_filestream‌(cls, filestream, index_keys, indexpath, log)‌ 331‌ ‌log‌.‌exception‌(‌"Can't read index file %r"‌,‌ ‌indexpath‌)‌ 332‌ ‌ --> 333‌ ‌return‌ ‌cls‌.‌from_filestream‌(‌filestream‌,‌ ‌index_keys‌)‌ 334‌ ‌ 335‌ ‌def‌ ‌iter‌(‌self‌)‌:‌

/usr/share/miniconda/envs/xarray-tests/lib/python3.7/site-packages/cfgrib/messages.py‌ in ‌from_filestream‌(cls, filestream, index_keys)‌ 262‌ ‌offsets‌ ‌=‌ ‌collections‌.‌OrderedDict‌(‌)‌ 263‌ ‌count_offsets‌ ‌=‌ ‌{‌}‌ ‌# type: T.Dict[int, int]‌ --> 264‌ ‌for‌ ‌message‌ ‌in‌ ‌filestream‌:‌ 265‌ ‌header_values‌ ‌=‌ ‌[‌]‌ 266‌ ‌for‌ ‌key‌ ‌in‌ ‌index_keys‌:‌

/usr/share/miniconda/envs/xarray-tests/lib/python3.7/site-packages/cfgrib/messages.py‌ in ‌iter‌(self)‌ 208‌ ‌def‌ ‌iter‌(‌self‌)‌:‌ 209‌ ‌# type: () -> T.Generator[Message, None, None]‌ --> 210‌ ‌with‌ ‌open‌(‌self‌.‌path‌,‌ ‌'rb'‌)‌ ‌as‌ ‌file‌:‌ 211‌ ‌valid_message_found‌ ‌=‌ ‌False‌ 212‌ ‌while‌ ‌True‌:‌

FileNotFoundError‌: [Errno 2] No such file or directory: '/home/vsts/work/1/s/doc/examples/data/era5-2mt-2019-03-uk.grib'‌ FileNotFoundError: [Errno 2] No such file or directory: '/home/vsts/work/1/s/doc/examples/data/era5-2mt-2019-03-uk.grib' ```

It appears that you are loading the data from a local file, rather than from the tutorial datasets api. This won't work, because the local file is not available in the build environment. That's why we added it to the tutorial repo in https://github.com/pydata/xarray-data/pull/14.

Can you try something like

python ds = xr.tutorial.load_dataset('era5-2mt-2019-03-uk.grib', engine='cfgrib')

Happy to add some markdown to this - that was an oversight. I'll work on that and push it up shortly.

In its current form, I'd say that your example is missing some text (markdown cells) which provide a narrative.

See this example - http://xarray.pydata.org/en/scipy19-docs/examples/ROMS_ocean_model.html - for what I am talking about.

Hi @zbruick - thanks for this and sorry for not following up sooner.

Do you feel this example is ready for review?