[03:02:58] <wikibugs>	 10Traffic, 10User-DannyS712: 404 error when using action=raw on an admin-level hidden revision - https://phabricator.wikimedia.org/T351688 (10DannyS712)
[03:03:25] <wikibugs>	 10Traffic, 10User-DannyS712: 404 error when using action=raw on an admin-level hidden revision - https://phabricator.wikimedia.org/T351688 (10DannyS712) If #traffic is not appropriate please let me know
[07:59:49] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10CodeReviewBot) vgutierrez opened https://gitlab.wikimedia.org/repos/sre/pybal/-/merge_requests/1  Release 1.15.14
[08:05:12] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10CodeReviewBot) vgutierrez merged https://gitlab.wikimedia.org/repos/sre/pybal/-/merge_requests/1  Release 1.15.14
[08:10:27] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10CodeReviewBot) vgutierrez opened https://gitlab.wikimedia.org/repos/sre/pybal/-/merge_requests/2  Release 1.15.14
[08:11:29] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10CodeReviewBot) vgutierrez opened https://gitlab.wikimedia.org/repos/sre/pybal/-/merge_requests/3  Release 1.15.14
[09:58:59] <jinxer-wm>	 (PuppetZeroResources) firing: Puppet has failed generate resources on ncredir2001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[10:13:59] <jinxer-wm>	 (PuppetZeroResources) resolved: Puppet has failed generate resources on ncredir2001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources
[11:22:34] <wikibugs>	 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host ms-fe2014.codfw.wmnet with OS bullseye
[11:53:55] <wikibugs>	 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:Install cp11[00-15] and rotate into production - https://phabricator.wikimedia.org/T349244 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by fabfur@cumin1001 for host cp1115.eqiad.wmnet with OS bullseye
[12:03:17] <wikibugs>	 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:Install cp11[00-15] and rotate into production - https://phabricator.wikimedia.org/T349244 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by fabfur@cumin1001 for host cp1115.eqiad.wmnet with OS bullseye executed with errors: - cp1115 (**FAIL*...
[12:03:34] <wikibugs>	 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:Install cp11[00-15] and rotate into production - https://phabricator.wikimedia.org/T349244 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by fabfur@cumin1001 for host cp1115.eqiad.wmnet with OS bullseye
[12:09:28] <wikibugs>	 10Traffic, 10SRE, 10SRE-swift-storage: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host ms-fe2014.codfw.wmnet with OS bullseye completed: - ms-fe2014 (**PASS**)   - Downtimed on Ici...
[12:38:38] <wikibugs>	 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:Install cp11[00-15] and rotate into production - https://phabricator.wikimedia.org/T349244 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by fabfur@cumin1001 for host cp1115.eqiad.wmnet with OS bullseye completed: - cp1115 (**PASS**)   - Remo...
[13:35:23] <wikibugs>	 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:Install cp11[00-15] and rotate into production - https://phabricator.wikimedia.org/T349244 (10Fabfur)
[14:44:20] <wikibugs>	 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:Install cp11[00-15] and rotate into production - https://phabricator.wikimedia.org/T349244 (10Fabfur)
[16:02:55] <wikibugs>	 10Traffic, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall)
[16:03:49] <wikibugs>	 10Traffic, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall)
[16:44:27] <duesen>	 vgutierrez: Coming back to the restbase CI issue... so far, CI on Github used to work with plain HTTP, no TLS at all. That never worked for me locally though. Is it possible that we have been whitelisting Github IPs for plain HTTP access? And now their addresses changed, and it broke?
[16:44:33] <duesen>	 hnowlan: --^
[16:46:03] <vgutierrez>	 ALLOW IPv4 80/tcp from 0.0.0.0/0
[16:46:36] <vgutierrez>	 duesen: port 80 is open to everbody on the Internet... maybe Github has deprecated HTTP?
[16:53:11] <taavi>	 the deployment-prep puppetmaster had for years a local commit with a message that was something like "[local hack] allow external http access to parsoid for ci purposes". i don't recall what it did but it's certainly isn't there anymore.
[17:24:41] <duesen>	 vgutierrez: I suppose it's MediaWiki itself that denies access without SSL... and has for a while when I tried locally. But somehow it used to work from Github...
[17:24:41] <duesen>	 Try curl -X POST -v -F "wikitext=test" -H "Host: en.wikipedia.beta.wmflabs.org" http://parsoid-external-ci.access.beta.wmflabs.org/api/rest_v1/transform/wikitext/to/html/Thorium
[17:25:32] <duesen>	 As opposed to curl --insecure -X POST -v -F "wikitext=test" -H "Host: en.wikipedia.beta.wmflabs.org" https://parsoid-external-ci.access.beta.wmflabs.org/api/rest_v1/transform/wikitext/to/html/Thorium
[17:26:10] <duesen>	 Anyway. I suppose the real solution is to make the tests use HTTPS, and use a cert that can actually be verified from the outside.
[17:27:45] <vgutierrez>	 ohhh
[17:28:09] <vgutierrez>	 that's definitely an unexpected side effect from moving port 80 from varnish to HAProxy
[17:28:57] <vgutierrez>	 yep.... traffic should use https as we don't support plain HTTP requests from the Internet
[18:32:29] <duesen>	 vgutierrez: thank you!
[18:54:50] <wikibugs>	 10Traffic, 10User-DannyS712: 404 error when using action=raw on an admin-level hidden revision - https://phabricator.wikimedia.org/T351688 (10Umherirrender) Seems expected that the revision-deleted content is not shown as there is a comment in `RawAction`  > // Public-only due to cache headers  But you right t...