[02:24:15] <wikibugs>	 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed (Feb 2022) - https://phabricator.wikimedia.org/T302699 (10AlexisJazz) 502 again.
[09:40:27] <wikibugs>	 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T305214 (10TheDJ)
[11:26:16] <wmf-insecte>	 Project beta-scap-sync-world build #48431: 04FAILURE in 1 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/48431/
[11:35:53] <wmf-insecte>	 Yippee, build fixed!
[11:35:53] <wmf-insecte>	 Project beta-scap-sync-world build #48432: 09FIXED in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/48432/
[11:50:46] <wikibugs>	 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T305214 (10TheDJ)
[14:02:29] <wikibugs>	 10Phabricator, 10User-AKlapper: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708 (10Aklapper)
[14:26:51] <wikibugs>	 10Phabricator, 10User-AKlapper: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708 (10Aklapper)
[14:56:20] <wmf-insecte>	 Project beta-scap-sync-world build #48452: 04FAILURE in 1 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/48452/
[15:05:55] <wmf-insecte>	 Yippee, build fixed!
[15:05:55] <wmf-insecte>	 Project beta-scap-sync-world build #48453: 09FIXED in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/48453/
[15:40:09] <wikibugs>	 10Project-Admins: Create project tag for <Add support for tracking specific namespaces to Programs & Events Dashboard> - https://phabricator.wikimedia.org/T306714 (10Enelesmai)
[16:41:07] <wikibugs>	 10Project-Admins: Create project tag for <Add support for tracking specific namespaces to Programs & Events Dashboard> - https://phabricator.wikimedia.org/T306714 (10Aklapper) 05Open→03Stalled Hi @Enelesmai, https://github.com/WikiEducationFoundation/WikiEduDashboard/issues exists for issue tracking and plan...
[19:51:20] <icinga-wm>	 PROBLEM - Gerrit Health Check on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus
[19:51:43] <icinga-wm>	 PROBLEM - Gerrit JSON on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus
[19:59:40] <AntiComposite>	 yup, can't load gerrit
[19:59:43] <wmf-insecte>	 Project beta-code-update-eqiad build #389131: 04FAILURE in 6 min 42 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/389131/
[20:00:17] <RhinosF1>	 AntiComposite: it's being looked at
[20:00:31] <RhinosF1>	 I pinged Re.edy who is trying to find someone
[20:03:10] <icinga-wm>	 RECOVERY - Gerrit JSON on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 42302 bytes in 0.113 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus
[20:04:19] <RhinosF1>	 Reedy: looks like load dropped back https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=gerrit1001&var-datasource=thanos&var-cluster=misc&viewPanel=3&from=now-30m&to=now
[20:04:22] <RhinosF1>	 Not all the way
[20:04:29] <RhinosF1>	 But enough to not make it die
[20:04:42] <wmf-insecte>	 Yippee, build fixed!
[20:04:42] <wmf-insecte>	 Project beta-code-update-eqiad build #389132: 09FIXED in 1 min 41 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/389132/
[20:04:44] <RhinosF1>	 Oh now it's gone below before
[20:05:04] <icinga-wm>	 RECOVERY - Gerrit Health Check on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 973 bytes in 0.040 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus
[22:12:31] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-Releasing: Have at least one job running REL1_XX with it's appropriate vendor repo - https://phabricator.wikimedia.org/T295175 (10Reedy) {T306721} being an example where this would have ben useful for #mw-1.38-release  CI was silently installing it when running patc...
[23:24:12] <icinga-wm>	 PROBLEM - Check systemd state on doc1001 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc1002.eqiad.wmnet.service,rsync-doc-doc2001.codfw.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state