[00:07:00] <jinxer-wm>	 FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[00:15:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[00:20:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[00:35:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[00:40:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[00:57:00] <jinxer-wm>	 RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[01:00:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[01:05:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[01:15:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[01:25:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[01:30:38] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[03:15:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[03:43:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[03:53:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[04:59:32] <wikibugs>	 (03PS24) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136)
[05:00:00] <wikibugs>	 (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[05:09:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[05:19:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[05:28:02] <wikibugs>	 (03PS25) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136)
[05:28:30] <wikibugs>	 (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[05:40:50] <wikibugs>	 (03PS1) 10Krinkle: Create template for CVNBot 5.x (pycvnbot) and monitor cvn-wp-nl [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332431 (https://phabricator.wikimedia.org/T327136)
[05:40:52] <wikibugs>	 (03PS1) 10Krinkle: Switch CVNBot11 to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332432 (https://phabricator.wikimedia.org/T327136)
[05:41:12] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Create template for CVNBot 5.x (pycvnbot) and monitor cvn-wp-nl [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332431 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[05:41:20] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Switch CVNBot11 to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332432 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[05:44:16] <wikibugs>	 (03PS1) 10Krinkle: build: Fix PHP 8.3 build [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332434
[05:44:56] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] build: Fix PHP 8.3 build [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332434 (owner: 10Krinkle)
[05:45:11] <wikibugs>	 (03PS2) 10Krinkle: Create template for CVNBot 5.x (pycvnbot) and monitor cvn-wp-nl [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332431 (https://phabricator.wikimedia.org/T327136)
[05:45:14] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] Create template for CVNBot 5.x (pycvnbot) and monitor cvn-wp-nl [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332431 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[05:45:17] <wikibugs>	 (03PS2) 10Krinkle: Switch CVNBot11 to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332432 (https://phabricator.wikimedia.org/T327136)
[05:45:20] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] Switch CVNBot11 to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332432 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[05:45:53] <wikibugs>	 (03Merged) 10jenkins-bot: build: Fix PHP 8.3 build [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332434 (owner: 10Krinkle)
[05:45:54] <wikibugs>	 (03Merged) 10jenkins-bot: Create template for CVNBot 5.x (pycvnbot) and monitor cvn-wp-nl [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332431 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[05:45:59] <wikibugs>	 (03Merged) 10jenkins-bot: Switch CVNBot11 to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332432 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[06:16:01] <wikibugs>	 (03PS26) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136)
[06:16:29] <wikibugs>	 (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[06:38:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[06:43:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[06:48:38] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[07:15:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[07:37:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[07:42:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[07:52:38] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[08:31:23] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Debian Bullseye deprecation extension for statanalyser - https://phabricator.wikimedia.org/T433570#12268577 (10Leaderboard) 05Open→03Resolved a:03Leaderboard We can resolve this for now as it's on Trixie.
[09:05:01] <wikibugs>	 (03update) 10countcount: Update cimg/rust Docker tag to v1.98 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/36 (owner: 10renovatebot)
[09:05:30] <wikibugs>	 (03update) 10countcount: Update cimg/rust Docker tag to v1.98 and fix pnpm bootstrap in the build job [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/36 (owner: 10renovatebot)
[09:08:05] <wikibugs>	 (03merge) 10countcount: Update cimg/rust Docker tag to v1.98 and fix pnpm bootstrap in the build job [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/36 (owner: 10renovatebot)
[09:55:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[10:00:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[10:34:26] <wikibugs>	 (03update) 10chimaifeanyi29: T432938: Add interface language switcher with i18n loading mechanism [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/9 (https://phabricator.wikimedia.org/T432938)
[10:46:40] <wikibugs>	 (03update) 10chimaifeanyi29: T435732: Add tooltip explanations for Mix, Editors, Edits [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/10 (https://phabricator.wikimedia.org/T435732)
[10:47:57] <wikibugs>	 (03update) 10chimaifeanyi29: T435732: Add tooltip explanations for Mix, Editors, Edits [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/10 (https://phabricator.wikimedia.org/T435732)
[11:13:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:15:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:18:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[13:54:09] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation): Migrate ocrtoy away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435779#12268796 (10Xover) 05Open→03Resolved a:03Xover Done. Code has been migrated to a new Trixie-based image and the old Bullseye-based instance has been shut do...
[14:38:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[15:15:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[16:18:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[16:32:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[16:37:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[16:38:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[16:48:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[17:36:30] <wikibugs>	 10Toolforge, 06tools-platform-team: maintain-harbor test suite should not output HTTP Authorization header - https://phabricator.wikimedia.org/T436249#12268899 (10Raymond_Ndibe) 05Open→03Resolved
[18:38:18] <wikibugs>	 (03merge) 10raymond-ndibe: Remove commit as an option in help message for `toolforge build start --ref` [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/149 (https://phabricator.wikimedia.org/T435793) (owner: 10sebastian-berlin-wmse)
[19:15:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[19:23:44] <wikibugs>	 06tools-platform-team, 10Toolhub: Add banner about Toolhub Evolved - https://phabricator.wikimedia.org/T434556#12268947 (10Schiste) @aputhin oh thank you so much that's great! Thanks (sorry I absolutely missed the previous mention!
[19:42:34] <wikibugs>	 (03update) 10raymond-ndibe: openapi: remove `rebuild_storage` from the interface [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/187 (https://phabricator.wikimedia.org/T434780) (owner: 10dcaro)
[20:00:37] <wikibugs>	 (03update) 10raymond-ndibe: openapi: remove `rebuild_storage` from the interface [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/187 (https://phabricator.wikimedia.org/T434780) (owner: 10dcaro)
[20:01:15] <wikibugs>	 (03approved) 10raymond-ndibe: openapi: remove `rebuild_storage` from the interface [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/187 (https://phabricator.wikimedia.org/T434780) (owner: 10dcaro)
[20:01:22] <wikibugs>	 (03merge) 10raymond-ndibe: openapi: remove `rebuild_storage` from the interface [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/187 (https://phabricator.wikimedia.org/T434780) (owner: 10dcaro)
[20:01:27] <wikibugs>	 (03update) 10raymond-ndibe: global: add public tool search with config endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/186 (https://phabricator.wikimedia.org/T433690) (owner: 10dcaro)
[20:05:07] <wikibugs>	 (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.210-20260830200138-c3fee900 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1393 (https://phabricator.wikimedia.org/T434780)
[20:12:32] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api
[20:17:14] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
[20:23:30] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-wikifeeds02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436462 (10Southparkfan) 03NEW
[20:23:52] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-schema-3 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436463 (10Southparkfan) 03NEW
[20:25:30] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-webperf2[12] (Bullseye deprecation) - https://phabricator.wikimedia.org/T436464 (10Southparkfan) 03NEW
[20:25:31] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-docker-mathoid02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436465 (10Southparkfan) 03NEW
[20:25:32] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-docker-citoid02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436466 (10Southparkfan) 03NEW
[20:25:34] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-docker-cxserver02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436467 (10Southparkfan) 03NEW
[20:31:46] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468 (10Southparkfan) 03NEW
[20:31:47] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-mwlog02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436469 (10Southparkfan) 03NEW
[20:31:49] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-cumin-3 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436470 (10Southparkfan) 03NEW
[20:31:52] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-sessionstore06 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436471 (10Southparkfan) 03NEW
[20:31:58] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace Acme-chief hosts: deployment-acme-chief0[56] (Bullseye deprecation - https://phabricator.wikimedia.org/T436472 (10Southparkfan) 03NEW
[20:32:14] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-cumin-3 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436470#12269125 (10Southparkfan)
[20:33:43] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace Acme-chief hosts: deployment-acme-chief0[56] (Bullseye deprecation) - https://phabricator.wikimedia.org/T436472#12269128 (10Southparkfan)
[20:34:47] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12269130 (10Southparkfan)
[20:36:40] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12269132 (10Southparkfan)
[20:36:42] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 07Epic, 06Release-Engineering-Team (Priority Backlog 📥): Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12269133 (10Southparkfan)
[20:37:05] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 07Epic, 06Release-Engineering-Team (Priority Backlog 📥): Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12269135 (10Southparkfan)
[20:45:15] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-etcd05 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436473 (10Southparkfan) 03NEW
[20:45:16] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-eventgate-4 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436474 (10Southparkfan) 03NEW
[20:46:51] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Data-Engineering, 10Event-Platform: Replace deployment-eventgate-4 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436474#12269167 (10Southparkfan)
[20:46:52] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Data-Engineering, 10Event-Platform: Replace deployment-eventstreams-2 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436475 (10Southparkfan) 03NEW
[20:46:53] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace Swift hosts: deployment-ms-(be0[78]|fe04) (Bullseye deprecation) - https://phabricator.wikimedia.org/T436476 (10Southparkfan) 03NEW
[20:46:57] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace or decommission RESTBase hosts: deployment-restbase(05|-bullseye) (Bullseye deprecation) - https://phabricator.wikimedia.org/T436477 (10Southparkfan) 03NEW
[20:47:50] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Data-Engineering, 10Event-Platform: Replace deployment-schema-3 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436463#12269218 (10Southparkfan)
[20:48:20] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Data-Engineering, 10Event-Platform: Replace deployment-eventgate-4 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436474#12269227 (10Southparkfan)
[20:48:20] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Data-Engineering, 10Event-Platform: Replace deployment-eventstreams-2 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436475#12269226 (10Southparkfan)
[20:48:22] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Data-Engineering, 10Event-Platform: Replace deployment-schema-3 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436463#12269228 (10Southparkfan)
[20:48:28] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12269225 (10Southparkfan)
[20:48:32] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 07Epic, 06Release-Engineering-Team (Priority Backlog 📥): Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12269229 (10Southparkfan)
[20:51:23] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api
[20:52:37] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 07Epic, 06Release-Engineering-Team (Priority Backlog 📥): Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12269234 (10Southparkfan)
[20:53:56] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Traffic: Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12269251 (10Southparkfan)
[20:54:13] <wikibugs>	 (03update) 10raymond-ndibe: models: support webservice job-type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898)
[20:55:57] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Doing 😎): Planning for a plan to migrate Beta Cluster again - https://phabricator.wikimedia.org/T433998#12269253 (10Southparkfan) >>! In T433998#12258671, @ssingh wrote: > Traffic will work on upgrading `dep...
[20:56:36] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
[20:58:06] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-echostore02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436479 (10Southparkfan) 03NEW
[20:58:16] <wikibugs>	 (03update) 10raymond-ndibe: components-api: bump to 0.0.210-20260830200138-c3fee900 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1393 (https://phabricator.wikimedia.org/T434780) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[20:58:17] <wikibugs>	 (03approved) 10raymond-ndibe: components-api: bump to 0.0.210-20260830200138-c3fee900 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1393 (https://phabricator.wikimedia.org/T434780) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[20:58:24] <wikibugs>	 (03merge) 10raymond-ndibe: components-api: bump to 0.0.210-20260830200138-c3fee900 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1393 (https://phabricator.wikimedia.org/T434780) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[20:58:42] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 07Epic, 06Release-Engineering-Team (Priority Backlog 📥): Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12269272 (10Southparkfan)
[21:03:44] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-rdb01 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436480 (10Southparkfan) 03NEW
[21:03:45] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-zookeeper-3 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436481 (10Southparkfan) 03NEW
[21:03:58] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 07Epic, 06Release-Engineering-Team (Priority Backlog 📥): Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12269299 (10Southparkfan)
[21:05:20] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Doing 😎): Planning for a plan to migrate Beta Cluster again - https://phabricator.wikimedia.org/T433998#12269303 (10Southparkfan) 05In progress→03Resolved For all services and hosts, subtasks have be...
[21:29:24] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-wikifeeds02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436462#12269311 (10Southparkfan) 05Open→03In progress a:03Southparkfan It wasn't too hard to get Wikifeeds up and running using [[ https://g...
[21:37:57] <wikibugs>	 (03PS27) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136)
[21:38:26] <wikibugs>	 (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[22:01:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:03:33] <wikibugs>	 (03update) 10raymond-ndibe: models: support webservice job-type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898)
[22:06:38] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:11:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:26:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:32:03] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-webperf2[12] (Bullseye deprecation) - https://phabricator.wikimedia.org/T436464#12269344 (10Southparkfan) 05Open→03In progress a:03Southparkfan
[22:37:09] <wikibugs>	 (03update) 10raymond-ndibe: global: add public tool search with config endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/186 (https://phabricator.wikimedia.org/T433690) (owner: 10dcaro)
[22:41:06] <wikibugs>	 (03approved) 10waldir: T435732: Add tooltip explanations for Mix, Editors, Edits [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/10 (https://phabricator.wikimedia.org/T435732) (owner: 10chimaifeanyi29)
[22:41:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:46:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:56:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:57:46] <wikibugs>	 06tools-platform-team: components-api: update toolforge models - https://phabricator.wikimedia.org/T436485 (10Raymond_Ndibe) 03NEW
[23:01:06] <wikibugs>	 (03open) 10raymond-ndibe: gen: update toolforge models [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/188 (https://phabricator.wikimedia.org/T436485)
[23:15:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: opentofu-infra-diff.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[23:40:12] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12269425 (10bd808) New hosts: * deployment-depl...