issue_comments

yeah, that makes the CI somewhat less useful. allow_failure is used for silencing the error code of pytest: https://github.com/pydata/xarray/blob/dd9fe2a8a414ddefa3b04b934163c9ccc628c5c7/ci/azure/unit-tests.yml#L14-L18

Not sure if that would make a difference, but maybe we should try using continueOnError instead?

Edit: we could also try to print a warning if the CI fails: yaml - bash: | $(environment_variables) pytest \ --junitxml=junit/test-results.xml \ --cov=xarray \ --cov-report=xml \ $(pytest_extra_flags) \ || ([ "$ALLOW_FAILURE" = "true" ] && echo -e "\043#vso[task.logissue type=warning;] ignored CI failure!!") see docs for the format of these messages. It seems they are supposed to be visible in the status report of github.

Another small thing, the py38-flaky is allowed to fail, which means that it is always green (unless someone checks the log).

https://github.com/pydata/xarray/blob/b76a13f042571d01ca07461f13125a030f7297ea/azure-pipelines.yml#L34

Thank you for the ping, @jhamman!

In a very ideal world, a failed nightly build would open an issue (or comment on an existing one) rather than send an email.

I concur with you Joe. I think this is 💯 doable today. I'm going to work on pushing it forward today, and will submit a PR once it's ready.

ping @andersy005 and @scottyhq who are the resident experts in the areas of github actions and automation.

In a very ideal world, a failed nightly build would open an issue (or comment on an existing one) rather than send an email. I'm sure this is not only possible, but already done elsewhere. Do you guys have thoughts on what would be doable here?

When thinking about this after we ended the call, I realized we would actually be losing a bit of coverage: what happens if we remove the upstream-dev CI for PRs and a PR introduces changes incompatible with upstream-dev? We would definitely catch that using the nightly CI, but only after merging.

That's a good point. A "allowed failure" would be even better.

Though on balance, assuming we can't do "allowed failures", I would favor more accurate test results over the immediacy of broken tests on upstream deps + new code. I'm not sure there are that many failures in the intersection of upstream deps + new code.

I think github actions have the option to send notifications for failed CI, so those who have that enabled would have to open an issue. Ideally, however, the failed action would automatically open a issue with a generic title (something like "nightly upstream-dev CI failed") and the build log (maybe trimmed to just the error logs?). Not sure if that is possible right now, though.

When thinking about this after we ended the call, I realized we would actually be losing a bit of coverage: what happens if we remove the upstream-dev CI for PRs and a PR introduces changes incompatible with upstream-dev? We would definitely catch that using the nightly CI, but only after merging.