html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/4574#issuecomment-727068025,https://api.github.com/repos/pydata/xarray/issues/4574,727068025,MDEyOklzc3VlQ29tbWVudDcyNzA2ODAyNQ==,14808389,2020-11-13T22:32:25Z,2020-11-13T22:49:49Z,MEMBER,"yeah, that makes the CI somewhat less useful. `allow_failure` is used for silencing the error code of `pytest`: https://github.com/pydata/xarray/blob/dd9fe2a8a414ddefa3b04b934163c9ccc628c5c7/ci/azure/unit-tests.yml#L14-L18
Not sure if that would make a difference, but maybe we should try using [`continueOnError`](https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema&viewFallbackFrom=vsts#job) instead?
Edit: we could also try to print a warning if the CI fails:
```yaml
- bash: |
$(environment_variables) pytest \
--junitxml=junit/test-results.xml \
--cov=xarray \
--cov-report=xml \
$(pytest_extra_flags) \
|| ([ ""$ALLOW_FAILURE"" = ""true"" ] && echo -e ""\043#vso[task.logissue type=warning;] ignored CI failure!!"")
```
see [docs](https://docs.microsoft.com/en-us/azure/devops/pipelines/scripts/logging-commands?view=azure-devops&tabs=bash) for the format of these messages. It seems they are supposed to be visible in the status report of github.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,741115905
https://github.com/pydata/xarray/issues/4574#issuecomment-726802553,https://api.github.com/repos/pydata/xarray/issues/4574,726802553,MDEyOklzc3VlQ29tbWVudDcyNjgwMjU1Mw==,10194086,2020-11-13T14:42:37Z,2020-11-13T14:42:37Z,MEMBER,"Another small thing, the `py38-flaky` is allowed to fail, which means that it is always green (unless someone checks the log).
https://github.com/pydata/xarray/blob/b76a13f042571d01ca07461f13125a030f7297ea/azure-pipelines.yml#L34","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,741115905
https://github.com/pydata/xarray/issues/4574#issuecomment-726234456,https://api.github.com/repos/pydata/xarray/issues/4574,726234456,MDEyOklzc3VlQ29tbWVudDcyNjIzNDQ1Ng==,13301940,2020-11-12T17:47:19Z,2020-11-12T17:47:29Z,MEMBER,"Thank you for the ping, @jhamman!
> In a very ideal world, a failed nightly build would open an issue (or comment on an existing one) rather than send an email.
I concur with you Joe. I think this is 💯 doable today. I'm going to work on pushing it forward today, and will submit a PR once it's ready. ","{""total_count"": 4, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 4, ""rocket"": 0, ""eyes"": 0}",,741115905
https://github.com/pydata/xarray/issues/4574#issuecomment-725843078,https://api.github.com/repos/pydata/xarray/issues/4574,725843078,MDEyOklzc3VlQ29tbWVudDcyNTg0MzA3OA==,2443309,2020-11-12T05:16:20Z,2020-11-12T05:16:20Z,MEMBER,"ping @andersy005 and @scottyhq who are the resident experts in the areas of github actions and automation.
In a very ideal world, a failed nightly build would open an issue (or comment on an existing one) rather than send an email. I'm sure this is not only possible, but already done elsewhere. Do you guys have thoughts on what would be doable here?","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,741115905
https://github.com/pydata/xarray/issues/4574#issuecomment-725795622,https://api.github.com/repos/pydata/xarray/issues/4574,725795622,MDEyOklzc3VlQ29tbWVudDcyNTc5NTYyMg==,5635139,2020-11-12T02:53:43Z,2020-11-12T02:53:43Z,MEMBER,"> When thinking about this after we ended the call, I realized we would actually be losing a bit of coverage: what happens if we remove the upstream-dev CI for PRs and a PR introduces changes incompatible with upstream-dev? We would definitely catch that using the nightly CI, but only _after_ merging.
That's a good point. A ""allowed failure"" would be even better.
Though on balance, assuming we can't do ""allowed failures"", I would favor more accurate test results over the immediacy of broken tests on upstream deps + new code. I'm not sure there are _that_ many failures in the intersection of upstream deps + new code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,741115905
https://github.com/pydata/xarray/issues/4574#issuecomment-725701318,https://api.github.com/repos/pydata/xarray/issues/4574,725701318,MDEyOklzc3VlQ29tbWVudDcyNTcwMTMxOA==,14808389,2020-11-11T22:40:52Z,2020-11-11T22:40:52Z,MEMBER,"I think github actions have the option to send notifications for failed CI, so those who have that enabled would have to open an issue. Ideally, however, the failed action would automatically open a issue with a generic title (something like ""nightly upstream-dev CI failed"") and the build log (maybe trimmed to just the error logs?). Not sure if that is possible right now, though.
When thinking about this after we ended the call, I realized we would actually be losing a bit of coverage: what happens if we remove the upstream-dev CI for PRs and a PR introduces changes incompatible with upstream-dev? We would definitely catch that using the nightly CI, but only *after* merging.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,741115905