[00:03:11] <wikibugs>	 (03open) 10raymond-ndibe: cli: add stop command [repos/cloud/toolforge/jobs-cli] (use_new_status_field) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/179 (https://phabricator.wikimedia.org/T436638)
[00:03:32] <wikibugs>	 (03update) 10raymond-ndibe: cli: add stop command [repos/cloud/toolforge/jobs-cli] (use_new_status_field) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/179 (https://phabricator.wikimedia.org/T436638)
[00:37:50] <wikibugs>	 10Toolforge, 06tools-platform-team, 13Patch-For-Review: jobs-api: support job stop/pause - https://phabricator.wikimedia.org/T436638#12278440 (10Raymond_Ndibe) 05Open→03In progress
[00:52:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[00:57:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[00:58:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[01:03:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[01:08:38] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[01:39:03] <wmcs-alerts>	 FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[01:41:59] <wmcs-alerts>	 FIRING: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[01:43:04] <wikibugs>	 10Tool-clip2commons: iOS WebKit form resets under memory pressure - https://phabricator.wikimedia.org/T434086#12278519 (10Kavaljeet_Singh) a:03Kavaljeet_Singh I am Claiming this task and working on it
[01:44:03] <wmcs-alerts>	 FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[01:46:59] <wmcs-alerts>	 RESOLVED: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[01:47:59] <wmcs-alerts>	 FIRING: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[01:49:03] <wmcs-alerts>	 RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[01:50:03] <wmcs-alerts>	 FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[01:55:03] <wmcs-alerts>	 RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[01:57:04] <wmcs-alerts>	 FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[01:59:52] <wikibugs>	 (03update) 10raymond-ndibe: Draft: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T348755 https://phabricator.wikimedia.org/T431421)
[02:00:39] <wikibugs>	 (03update) 10raymond-ndibe: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T348755 https://phabricator.wikimedia.org/T431421)
[02:02:04] <wmcs-alerts>	 RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[02:02:14] <wmcs-alerts>	 RESOLVED: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[02:04:03] <wikibugs>	 10Tool-i18nsearch: duplicate messages when searching without specifying a repo - https://phabricator.wikimedia.org/T436697#12278542 (10jsn.sherman) 05Open→03Resolved Should be fixed now, @Michael, please reopen if you are still seeing it after a hard reload.
[02:04:54] <wikibugs>	 (03open) 10kavaljeetsingh: Fix iOS WebKit form resets under memory pressure (T434086) [toolforge-repos/clip2commons] - 10https://gitlab.wikimedia.org/toolforge-repos/clip2commons/-/merge_requests/12
[02:09:22] <wikibugs>	 (03open) 10raymond-ndibe: core, api: add job stop support [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/377 (https://phabricator.wikimedia.org/T436638)
[02:10:22] <wikibugs>	 (03update) 10raymond-ndibe: cli: add stop command [repos/cloud/toolforge/jobs-cli] (use_new_status_field) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/179 (https://phabricator.wikimedia.org/T436638)
[03:03:44] <wikibugs>	 10Tool-clip2commons: iOS WebKit form resets under memory pressure - https://phabricator.wikimedia.org/T434086#12278610 (10Kavaljeet_Singh) https://gitlab.wikimedia.org/toolforge-repos/clip2commons/-/merge_requests/12
[04:03:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[04:06:24] <wikibugs>	 (03PS1) 10Krinkle: docs: Update runbook for CVNBot python and rewrite in markdown [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1333458 (https://phabricator.wikimedia.org/T395164)
[04:18:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[04:23:38] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[05:03:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[05:08:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[05:14:24] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17
[05:14:32] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16
[05:14:44] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29
[05:14:50] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28
[05:15:03] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24
[05:15:10] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23
[05:16:08] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56
[05:16:13] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55
[05:17:13] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41
[05:17:47] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40
[05:18:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[05:23:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[05:24:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[05:34:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[05:56:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[06:01:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[06:26:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[06:30:39] <wikibugs>	 (03PS1) 10Krinkle: Switch CVNClerkBot to cvn-app15 [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1333553 (https://phabricator.wikimedia.org/T395164)
[06:31:40] <wikibugs>	 (03PS2) 10Krinkle: Switch CVNClerkBot to cvn-app15 [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1333553 (https://phabricator.wikimedia.org/T395164)
[06:35:35] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] Switch CVNClerkBot to cvn-app15 [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1333553 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[06:36:07] <wikibugs>	 (03Merged) 10jenkins-bot: Switch CVNClerkBot to cvn-app15 [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1333553 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[06:40:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[06:45:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[06:45:51] <wikibugs>	 (03PS2) 10Krinkle: docs: Update runbook for CVNBot python, rewrite in markdown [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1333458 (https://phabricator.wikimedia.org/T395164)
[06:46:00] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] docs: Update runbook for CVNBot python, rewrite in markdown [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1333458 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[06:48:59] <wikibugs>	 (03PS1) 10Krinkle: Switch remaining bots to CVNBot 5 (pycvnbot) [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1333568 (https://phabricator.wikimedia.org/T327136)
[06:49:50] <wikibugs>	 (03CR) 10Krinkle: [V:03+2 C:03+2] docs: Update runbook for CVNBot python, rewrite in markdown [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1333458 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[06:51:04] <wikibugs>	 06tools-infrastructure-team: Upgrade cloudcumin hosts to Trixie - https://phabricator.wikimedia.org/T436667#12278831 (10MoritzMuehlenhoff)
[06:51:13] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] Switch remaining bots to CVNBot 5 (pycvnbot) [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1333568 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[06:52:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[06:52:50] <wikibugs>	 (03Merged) 10jenkins-bot: Switch remaining bots to CVNBot 5 (pycvnbot) [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1333568 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[06:57:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[07:02:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[07:07:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[07:08:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[07:13:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[07:14:11] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17
[07:14:12] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16
[07:14:19] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29
[07:14:20] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28
[07:14:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[07:14:42] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24
[07:14:46] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23
[07:15:49] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56
[07:15:50] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55
[07:16:29] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41
[07:16:30] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40
[07:24:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[07:32:12] <wikibugs>	 (03PS1) 10Krinkle: Move all cvn-app12 tasks to cvn-app15 [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1333592 (https://phabricator.wikimedia.org/T395164)
[07:33:14] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] Move all cvn-app12 tasks to cvn-app15 [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1333592 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[07:33:51] <wikibugs>	 (03Merged) 10jenkins-bot: Move all cvn-app12 tasks to cvn-app15 [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1333592 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[07:54:00] <wikibugs>	 (03PS1) 10Krinkle: Move all cvn-app12 tasks to cvn-app16 [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1333620 (https://phabricator.wikimedia.org/T395164)
[07:54:51] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] Move all cvn-app12 tasks to cvn-app16 [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1333620 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[07:55:27] <wikibugs>	 (03Merged) 10jenkins-bot: Move all cvn-app12 tasks to cvn-app16 [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1333620 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[07:55:56] <wikibugs>	 (03PS1) 10Krinkle: docs: Write "Move all bots" section [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1333655 (https://phabricator.wikimedia.org/T395164)
[08:02:36] <wikibugs>	 (03PS2) 10Krinkle: docs: Write "Move all bots" section [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1333655 (https://phabricator.wikimedia.org/T395164)
[08:04:33] <wikibugs>	 06cloud-services-team, 10Data-Services, 10Toolforge, 06tools-platform-team, 07Documentation: Restructure and improve content for: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database - https://phabricator.wikimedia.org/T232404#12279030 (10RAdinsatoshi2008)
[08:04:46] <wikibugs>	 06cloud-services-team, 10Toolforge, 06tools-platform-team, 07Documentation: Update and Improve Toolforge and Cloud VPS Technical Documentation - https://phabricator.wikimedia.org/T203131#12279032 (10RAdinsatoshi2008)
[08:08:10] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Countervandalism-Network: Upgrade cvn fleet from Debian 11 Bullseye to Debian 13 Trixie (PHP 7.4 > PHP 8.4) - https://phabricator.wikimedia.org/T395164#12279040 (10Krinkle) 05Open→03Resolved
[08:11:45] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Countervandalism-Network: Upgrade cvn fleet from Debian 11 Bullseye to Debian 13 Trixie - https://phabricator.wikimedia.org/T395164#12279055 (10Krinkle)
[08:12:31] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Countervandalism-Network: Upgrade cvn fleet from Debian 11 Bullseye to Debian 13 Trixie - https://phabricator.wikimedia.org/T395164#12279057 (10Krinkle)
[08:14:31] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17
[08:14:38] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16
[08:14:52] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29
[08:14:58] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28
[08:15:04] <wmcs-alerts>	 FIRING: [2x] InstanceDown: Project cvn instance cvn-app10 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[08:15:04] <wmcs-alerts>	 FIRING: WidespreadInstanceDown: Widespread instances down in project cvn   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadInstanceDown
[08:15:11] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24
[08:15:17] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23
[08:16:17] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56
[08:16:23] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55
[08:17:23] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41
[08:17:59] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40
[08:20:04] <wmcs-alerts>	 RESOLVED: [2x] InstanceDown: Project cvn instance cvn-app10 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[08:20:04] <wmcs-alerts>	 RESOLVED: WidespreadInstanceDown: Widespread instances down in project cvn   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadInstanceDown
[08:34:35] <wikibugs>	 (03CR) 10Krinkle: [V:03+2 C:03+2] docs: Write "Move all bots" section [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1333655 (https://phabricator.wikimedia.org/T395164) (owner: 10Krinkle)
[08:36:42] <wikibugs>	 (03update) 10fnegri: alerts-api: Add initial scaffolding [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T435971)
[08:36:43] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: Add scaffolding for the Toolforge Alerting Service - https://phabricator.wikimedia.org/T435971#12279177 (10fnegri)
[08:38:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[08:39:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[08:44:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[08:47:33] <wikibugs>	 10Tool-wikinewsie, 10LPL sprints, 06translatewiki.net, 10LPL Projects (Ongoing maintenance): Add Wikinewsie to translatewiki.net - https://phabricator.wikimedia.org/T435739#12279213 (10Nikerabbit)
[08:48:31] <wikibugs>	 10Tool-wikinewsie, 10LPL sprints, 06translatewiki.net, 07Essential-Work, 10LPL Projects (Ongoing maintenance): Add Wikinewsie to translatewiki.net - https://phabricator.wikimedia.org/T435739#12279216 (10Nikerabbit)
[08:59:47] <wikibugs>	 (03CR) 10Hashar: "Yeah that is fair, at least once the code base is using Python that opens the path to port to EventStreams :-]" [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[09:14:47] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17
[09:14:47] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16
[09:14:55] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29
[09:14:56] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28
[09:15:03] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24
[09:15:04] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23
[09:15:59] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56
[09:16:00] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55
[09:16:41] <wikibugs>	 (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41
[09:16:42] <wikibugs>	 (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40
[09:17:46] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[09:22:46] <jinxer-wm>	 RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[09:27:00] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 project-proxy START - Cookbook wmcs.vps.remove_instance for instance maps-proxy-5
[09:27:47] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 project-proxy END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance maps-proxy-5
[09:28:42] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 project-proxy START - Cookbook wmcs.vps.remove_instance for instance maps-proxy-6
[09:28:58] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 project-proxy END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance maps-proxy-6
[10:49:13] <wikibugs>	 (03open) 10samwilson: Upgrade to Symfony 8.1 [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/12
[10:49:19] <wikibugs>	 (03update) 10samwilson: Draft: Upgrade to Symfony 8.1 [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/12
[10:57:00] <wikibugs>	 (03update) 10samwilson: Draft: Upgrade to Symfony 8.1 [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/12
[11:01:04] <wmcs-alerts>	 FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[11:02:58] <wikibugs>	 (03update) 10samwilson: Draft: Upgrade to Symfony 8.1 [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/12
[11:06:04] <wmcs-alerts>	 RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[11:16:04] <wmcs-alerts>	 FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[11:21:04] <wmcs-alerts>	 RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[11:21:59] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-4
[11:22:50] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-4
[11:23:10] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-5
[11:24:00] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-5
[11:24:22] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-6
[11:24:41] <wikibugs>	 (03update) 10taavi: tools: Remove Elasticsearch DNS record [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/125 (https://phabricator.wikimedia.org/T401818)
[11:24:43] <wikibugs>	 (03update) 10taavi: tools: Remove Elasticsearch volumes [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/126 (https://phabricator.wikimedia.org/T401818)
[11:24:59] <wikibugs>	 (03open) 10taavi: tools: Remove Elasticsearch DNS record [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/125 (https://phabricator.wikimedia.org/T401818)
[11:24:59] <wikibugs>	 (03open) 10taavi: tools: Remove Elasticsearch volumes [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/126 (https://phabricator.wikimedia.org/T401818)
[11:25:12] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-6
[11:27:53] <wikibugs>	 (03update) 10taavi: tools: Remove Elasticsearch DNS record [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/125 (https://phabricator.wikimedia.org/T401818)
[11:30:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:34:06] <wikibugs>	 (03open) 10taavi: eqiad1: project-proxy: Remove static proxy security groups [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/348 (https://phabricator.wikimedia.org/T431284)
[11:34:12] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/348
[11:34:37] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/348
[11:35:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[11:35:59] <wmcs-alerts>	 FIRING: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[11:36:04] <wmcs-alerts>	 FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[11:40:59] <wmcs-alerts>	 RESOLVED: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[11:41:04] <wmcs-alerts>	 RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[11:50:45] <wikibugs>	 (03update) 10samwilson: Draft: Upgrade to Symfony 8.1 [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/12
[11:54:37] <wikibugs>	 (03update) 10samwilson: Draft: Upgrade to Symfony 8.1 [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/12
[11:59:59] <wmcs-alerts>	 FIRING: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[12:00:03] <wmcs-alerts>	 FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[12:03:44] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12280181 (10BTullis) I have scheduled a backport deployment of the eventstre...
[12:04:59] <wmcs-alerts>	 RESOLVED: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[12:05:03] <wmcs-alerts>	 RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[12:19:10] <wikibugs>	 (03update) 10samwilson: Draft: Upgrade to Symfony 8.1 [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/12
[12:22:16] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12280316 (10BTullis) Canary testing with Wikimedia Debug extension looks goo...
[12:27:23] <wikibugs>	 10Tool-clip2commons: iOS WebKit form resets under memory pressure - https://phabricator.wikimedia.org/T434086#12280339 (10Dev) Thanks for taking this task @Kavaljeet_Singh. I will review your patch soon and let you know if there's anything.
[12:30:49] <wikibugs>	 (03update) 10samwilson: Draft: Upgrade to Symfony 8.1 [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/12
[12:38:56] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[12:45:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[12:46:14] <wikibugs>	 10Tool-i18nsearch: duplicate messages when searching without specifying a repo - https://phabricator.wikimedia.org/T436697#12280449 (10Michael) At first glance it looks good and working to me. Thanks!
[12:50:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[12:55:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[12:55:54] <wikibugs>	 (03approved) 10tlepage: alerts-api: Add initial scaffolding [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T435971) (owner: 10fnegri)
[12:58:57] <wikibugs>	 (03update) 10tlepage: alerts-api: Add initial scaffolding [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T435971) (owner: 10fnegri)
[13:00:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[13:05:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[13:15:53] <wikibugs>	 (03PS1) 10Hashar: Add .gitreview file [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333842
[13:19:04] <wmcs-alerts>	 FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[13:22:38] <wikibugs>	 06tools-platform-team: [all] first of the month dependency upgrades for september - https://phabricator.wikimedia.org/T436796 (10Raymond_Ndibe) 03NEW
[13:24:04] <wmcs-alerts>	 RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[13:25:03] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] Add .gitreview file [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333842 (owner: 10Hashar)
[13:26:58] <wikibugs>	 (03Merged) 10jenkins-bot: Add .gitreview file [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333842 (owner: 10Hashar)
[13:27:38] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/76 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:31:32] <wikibugs>	 10Cloud-VPS (Project-requests), 06Data-Platform-SRE: Request creation of cephvmm VPS project - https://phabricator.wikimedia.org/T436647#12280714 (10Andrew) We don't typically offer bare-metal cinder volumes, only ceph rbd volumes. I'm a bit worried that...  a) running ceph on ceph will overload the ceph backe...
[13:41:14] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/75 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:41:15] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/106 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:41:16] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/14 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:41:17] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/69 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:41:19] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/341 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:41:23] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/119 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:41:27] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/94 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:41:31] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/375 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:41:35] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/22 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:41:39] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/105 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:41:43] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/167 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:41:47] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/78 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:41:51] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/151 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:41:55] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1395 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:41:59] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/60 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:42:03] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/52 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:42:07] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/35 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:42:11] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/118 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:42:16] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/374 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:42:20] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/190 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:42:24] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/121 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:42:28] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/150 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:42:32] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/97 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:42:36] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/59 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:42:40] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:42:44] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/43 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:42:52] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/43 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:42:56] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/93 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:43:00] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/177 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:43:04] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/189 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:43:08] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/96 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:43:12] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/22 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:43:16] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/36 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:44:13] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/36 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:44:22] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/36 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:44:31] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/36 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:44:32] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/22 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:47:21] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/22 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:47:29] <wikibugs>	 (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.46-20260902134448-5ad1fe9e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1396
[13:47:29] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.46-20260902134448-5ad1fe9e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1396
[13:47:32] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/22 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:48:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[13:49:15] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/96 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:49:16] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/96 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:49:21] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/189 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:49:21] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/96 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:49:23] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/189 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:49:27] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/177 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:49:30] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/177 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:49:30] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/189 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:49:38] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/93 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:49:42] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/177 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:49:46] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/93 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:49:50] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/93 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:49:54] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/43 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:49:58] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/43 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:50:02] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/43 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:50:06] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/43 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:50:10] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/43 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:50:14] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:50:18] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/43 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:50:22] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:50:26] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/59 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:50:35] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/59 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:50:39] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:50:43] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/97 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:50:47] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/97 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:50:51] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/150 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:50:55] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/59 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:50:59] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/150 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:51:03] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/121 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:51:07] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/121 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:51:15] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/150 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:51:19] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/190 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:51:35] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/190 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:51:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[13:51:40] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/121 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:51:44] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/374 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:51:48] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/374 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:52:04] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/118 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:52:08] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/118 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:52:12] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/374 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:52:20] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/35 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:52:24] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/35 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:52:28] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/118 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:52:37] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/52 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:52:41] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/52 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:52:45] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/35 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:52:49] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/60 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:52:53] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/60 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:52:57] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/52 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:53:01] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1395 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:53:09] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1395 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:53:13] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/151 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:53:17] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/151 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:53:25] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1395 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:53:30] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/151 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:53:34] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/78 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:53:38] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/78 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:53:42] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/167 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:53:46] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/78 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:53:50] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/167 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:53:58] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/105 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:54:02] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/105 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:54:06] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/22 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:54:10] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/167 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:54:14] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/22 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:54:22] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/105 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:54:30] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/375 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:54:38] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/375 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:54:42] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/22 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:54:46] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/94 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:54:53] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/94 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:55:01] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/119 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:55:05] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/375 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:55:09] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/119 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:55:13] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/94 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:55:21] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/341 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:55:29] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/341 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:55:33] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/69 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:55:37] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/14 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:55:42] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/341 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:55:46] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/69 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:55:49] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/14 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:55:54] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/69 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:55:58] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/106 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:56:02] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/14 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:56:06] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/106 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:56:10] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/75 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:56:14] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/75 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:56:22] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/76 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:56:30] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/106 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:56:34] <wikibugs>	 (03approved) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/76 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:56:42] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/75 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:56:50] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/76 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:57:03] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:57:07] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/97 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:57:11] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/190 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:57:15] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/60 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:57:23] <wikibugs>	 (03update) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/119 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:57:47] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/76 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:57:51] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/97 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:58:17] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/119 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:58:31] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/190 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:58:45] <wikibugs>	 (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.567-20260902135207-ed885f8b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1397
[13:58:49] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.567-20260902135207-ed885f8b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1397
[13:59:08] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/60 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[13:59:38] <wikibugs>	 (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: ingress-admission: bump to 0.0.90-20260902135022-e09fd871 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1398
[13:59:56] <wikibugs>	 06tools-platform-team: [all] first of the month dependency upgrades for september - https://phabricator.wikimedia.org/T436796#12280930 (10Raymond_Ndibe) 05Open→03In progress
[14:00:39] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.47-20260902135501-fabf4256 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1396
[14:00:48] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.47-20260902135501-fabf4256 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1396
[14:01:35] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: calico: bump to 0.0.20-20260902135340-31d4b17e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1399
[14:01:36] <wikibugs>	 (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: calico: bump to 0.0.20-20260902135340-31d4b17e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1399
[14:01:38] <wmcs-alerts>	 RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[14:02:28] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/106 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[14:02:56] <wikibugs>	 (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.93-20260902135126-cc374cbd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1400
[14:03:01] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.93-20260902135126-cc374cbd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1400
[14:03:24] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/151 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[14:04:04] <wikibugs>	 10Tool-Pageviews, 06Data-Engineering, 06Data-Engineering-Icebox: Pageviews Analysis 2026 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#12280946 (10MusikAnimal)
[14:04:15] <wikibugs>	 (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.212-20260902135855-3e34d704 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1401
[14:04:17] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.212-20260902135855-3e34d704 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1401
[14:05:25] <wikibugs>	 (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-builder: bump to 0.0.153-20260902135338-e7c94e34 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1402
[14:05:25] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.94-20260902135935-991a08d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1400
[14:05:27] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-builder: bump to 0.0.153-20260902135338-e7c94e34 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1402
[14:05:36] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.94-20260902135935-991a08d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1400
[14:08:58] <wikibugs>	 (03PS1) 10Hashar: build: ensure CI passes against each requests python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333865
[14:11:54] <wikibugs>	 (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-kubeusers: bump to 0.0.202-20260902134959-47547230 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1403
[14:11:57] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-kubeusers: bump to 0.0.202-20260902134959-47547230 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1403
[14:12:25] <wikibugs>	 (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: volume-admission: bump to 0.0.88-20260902135156-b9eeaaf1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1404
[14:12:27] <wikibugs>	 (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-api: bump to 0.0.221-20260902135258-7a258309 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1405
[14:12:29] <wikibugs>	 (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.106-20260902135305-9d2ad173 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1406
[14:12:35] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: volume-admission: bump to 0.0.88-20260902135156-b9eeaaf1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1404
[14:12:36] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-api: bump to 0.0.221-20260902135258-7a258309 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1405
[14:12:40] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.106-20260902135305-9d2ad173 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1406
[14:14:26] <wikibugs>	 (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: envvars-admission: bump to 0.0.42-20260902135146-45240b57 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1407
[14:14:26] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: envvars-admission: bump to 0.0.42-20260902135146-45240b57 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1407
[14:16:03] <wikibugs>	 (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: registry-admission: bump to 0.0.79-20260902140719-e192d371 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1408
[14:16:17] <wikibugs>	 (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-harbor: bump to 0.0.74-20260902135358-6d48c550 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1409
[14:16:18] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-harbor: bump to 0.0.74-20260902135358-6d48c550 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1409
[14:16:32] <wikibugs>	 (03merge) 10raymond-ndibe: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/375 (https://phabricator.wikimedia.org/T436796) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce)
[14:17:56] <wikibugs>	 (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: wmcs-k8s-metrics: bump to 0.0.25-20260902141113-35f68462 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1410
[14:18:50] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.107-20260902140532-1881747b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1406
[14:18:55] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.107-20260902140532-1881747b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1406
[14:21:13] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.568-20260902141641-99c9c4e5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1397
[14:21:16] <wikibugs>	 (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.568-20260902141641-99c9c4e5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1397
[14:23:56] <wikibugs>	 10Toolforge, 06tools-platform-team: [all] first of the month dependency upgrades for september - https://phabricator.wikimedia.org/T436796#12281061 (10taavi)
[14:25:05] <wikibugs>	 (03merge) 10taavi: eqiad1: project-proxy: Remove static proxy security groups [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/348 (https://phabricator.wikimedia.org/T431284)
[14:25:06] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
[14:26:11] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Fold special maps proxies back to the normal proxy - https://phabricator.wikimedia.org/T431284#12281071 (10taavi) 05Open→03Resolved
[14:26:23] <logmsgbot_cloud>	 !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
[14:41:45] <wikibugs>	 10Cloud-VPS (Project-requests), 06Data-Platform-SRE: Request creation of cephvmm VPS project - https://phabricator.wikimedia.org/T436647#12281179 (10LSobanski) Are there any particular concerns about using Ganeti that drive testing alternative solutions? I'm asking as I'm concerned about introducing additional...
[14:44:21] <wikibugs>	 (03PS1) 10Hashar: build: fix tox config for linting [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333881
[14:44:28] <wikibugs>	 10Cloud-VPS (Project-requests), 06Data-Platform-SRE: Request creation of cephvmm VPS project - https://phabricator.wikimedia.org/T436647#12281189 (10Gehel) Let's put this request on hold for the moment. We need to make some more progress on the planning / strategy first.  To add some context:  * This is about...
[14:44:48] <wikibugs>	 (03PS2) 10Hashar: build: ensure CI passes against each requested python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333865
[14:44:57] <wikibugs>	 (03PS2) 10Hashar: build: fix tox config for linting [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333881
[14:45:58] <wikibugs>	 (03CR) 10CI reject: [V:04-1] build: fix tox config for linting [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333881 (owner: 10Hashar)
[14:49:16] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] build: ensure CI passes against each requested python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333865 (owner: 10Hashar)
[14:49:41] <wikibugs>	 (03PS3) 10Hashar: build: fix tox config for linting [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333881
[14:49:49] <wikibugs>	 (03CR) 10Krinkle: build: fix tox config for linting (031 comment) [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333881 (owner: 10Hashar)
[14:51:02] <wikibugs>	 (03Merged) 10jenkins-bot: build: ensure CI passes against each requested python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333865 (owner: 10Hashar)
[14:54:21] <wikibugs>	 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-api] Implement an endpoint to retrieve system-level event logs - https://phabricator.wikimedia.org/T432566#12281287 (10TLepage-WMF)
[14:55:23] <wikibugs>	 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12281303 (10VRiley-WMF) 05Open→03In progress Thanks, I am proceeding with this now
[14:57:38] <wikibugs>	 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12281323 (10VRiley-WMF) adding this for record keepin on steps  Physically relocate the host now. Update Netbox Update N...
[14:57:44] <wikibugs>	 10Cloud-VPS (Project-requests), 06Data-Platform-SRE: Request creation of cephvmm VPS project - https://phabricator.wikimedia.org/T436647#12281324 (10bking) @LSobanski I agree 100% with your concerns, and I just want to make clear we won't be introducing anything anytime soon, and not without buy-in from multip...
[14:59:38] <wikibugs>	 (03CR) 10Hashar: build: fix tox config for linting (031 comment) [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333881 (owner: 10Hashar)
[15:03:03] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12281342 (10BTullis) The new eventstreams are being ingested by gobblin. {F1...
[15:09:45] <wikibugs>	 (03PS4) 10Hashar: build: fix tox config for linting [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333881
[15:10:36] <wikibugs>	 (03PS2) 10Hashar: Add coverage report [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333863
[15:10:54] <wikibugs>	 (03CR) 10Krinkle: [C:03+1] "Copied votes on follow-up patch sets have been updated:" (031 comment) [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333881 (owner: 10Hashar)
[15:16:33] <wikibugs>	 (03merge) 10apaskulin: Edit index.md to replace RapiDoc link that goes to a dead/hacked page [toolforge-repos/api-spec-reader] - 10https://gitlab.wikimedia.org/toolforge-repos/api-spec-reader/-/merge_requests/3 (owner: 10tburmeister)
[15:22:10] <wikibugs>	 06cloud-services-team, 10Tools, 07Performance Issue: Stalktoy often slow or unresponsive due to heavy traffic - https://phabricator.wikimedia.org/T427941#12281540 (10Pathoschild) 05Open→03Resolved p:05Triage→03High a:03Pathoschild
[15:41:26] <wikibugs>	 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12281636 (10BTullis)
[15:54:04] <wikibugs>	 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12281684 (10VRiley-WMF) Starting the first three  cloudvirt1067 D5 - U09 CableID: 5311 Port: 7  cloudvirt1066 D5 - U08 C...
[16:02:59] <wmcs-alerts>	 FIRING: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[16:03:03] <wmcs-alerts>	 FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[16:07:59] <wmcs-alerts>	 RESOLVED: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[16:08:03] <wmcs-alerts>	 RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[16:21:49] <wikibugs>	 (03PS1) 10Krinkle: Install oidentd on cvn-app to speed up Libera Chat connections [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1333923 (https://phabricator.wikimedia.org/T395274)
[16:22:14] <wikibugs>	 (03CR) 10Krinkle: [V:03+2 C:03+2] "Thanks Taavi!" [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1333923 (https://phabricator.wikimedia.org/T395274) (owner: 10Krinkle)
[16:29:00] <wikibugs>	 (03update) 10ameisenigel: Add `import` user group [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/6 (https://phabricator.wikimedia.org/T263081)
[16:29:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[16:29:45] <wikibugs>	 (03PS1) 10Krinkle: ProjectList: Remove need for "snamespaces" field [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333927
[16:30:25] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] build: fix tox config for linting [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333881 (owner: 10Hashar)
[16:32:04] <wikibugs>	 (03Merged) 10jenkins-bot: build: fix tox config for linting [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333881 (owner: 10Hashar)
[16:34:47] <wikibugs>	 (03merge) 10ameisenigel: Add `import` user group [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/6 (https://phabricator.wikimedia.org/T263081)
[16:35:41] <wikibugs>	 (03PS2) 10Krinkle: ProjectList: Remove need for "snamespaces" field [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333927
[16:39:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[16:47:08] <wikibugs>	 (03PS3) 10Krinkle: ProjectList: Remove need for "snamespaces" field [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333927
[16:50:49] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Replace deployment-schema-3 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436463#12281978 (10Ahoelzl) a:03tchin
[16:51:26] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Replace deployment-eventgate-4 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436474#12281989 (10Ahoelzl) a:03tchin
[16:51:30] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Replace deployment-schema-3 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436463#12281993 (10Ottomata) →14Duplicate dup:03T429497
[16:51:32] <wikibugs>	 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12281995 (10Ottomata)
[16:52:19] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12282004 (10taavi)
[16:52:22] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12282005 (10Ottomata)
[16:54:25] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Data-Engineering, 10Event-Platform: Replace deployment-eventstreams-2 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436475#12282035 (10Ottomata) →14Duplicate dup:03T429497
[16:54:28] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12282039 (10Ottomata)
[16:54:28] <wikibugs>	 (03PS1) 10Krinkle: Project: Fix generate_regex to create only one `<item1>` per message [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333946 (https://phabricator.wikimedia.org/T327136)
[16:55:06] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Replace deployment-eventgate-4 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436474#12282050 (10Ottomata) →14Duplicate dup:03T429497
[16:55:10] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12282052 (10Ottomata)
[16:55:48] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Project: Fix generate_regex to create only one `<item1>` per message [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333946 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[16:56:19] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12282067 (10Ottomata) @Southparkfan thanks for the tasks! For ease of project...
[16:59:46] <wikibugs>	 06cloud-services-team, 10Tool-paulina, 10Toolforge: Automated traffic affects paulina.toolforge.org performance - https://phabricator.wikimedia.org/T432878#12282124 (10Pepe_piton) In case it is useful for botnet detection, looking at the Grafana chart I detect that each day, at 4 PM UTC o'clock, the traf...
[17:00:52] <wikibugs>	 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12282130 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1...
[17:02:51] <wikibugs>	 (03PS2) 10Krinkle: Project: Fix generate_regex to create only one `<item1>` per message [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333946 (https://phabricator.wikimedia.org/T327136)
[17:02:51] <wikibugs>	 (03PS1) 10Krinkle: Project: Remove unneeded `speciallog` from Projects.xml [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333951
[17:03:27] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12282153 (10dancy) `wmf-beta-update-all` is currently failing due to:...
[17:04:17] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] "`tox -e cover && open htmlcov/index.html` . Works for me!" [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333863 (owner: 10Hashar)
[17:04:49] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] ProjectList: Remove need for "snamespaces" field [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333927 (owner: 10Krinkle)
[17:04:54] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] Project: Fix generate_regex to create only one `<item1>` per message [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333946 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[17:06:20] <wikibugs>	 (03Merged) 10jenkins-bot: ProjectList: Remove need for "snamespaces" field [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333927 (owner: 10Krinkle)
[17:06:23] <wikibugs>	 (03Merged) 10jenkins-bot: Project: Fix generate_regex to create only one `<item1>` per message [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333946 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle)
[17:06:23] <wikibugs>	 (03PS2) 10Krinkle: Project: Remove unneeded `speciallog` from Projects.xml [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333951
[17:06:30] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] Project: Remove unneeded `speciallog` from Projects.xml [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333951 (owner: 10Krinkle)
[17:06:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[17:08:22] <wikibugs>	 (03Merged) 10jenkins-bot: Project: Remove unneeded `speciallog` from Projects.xml [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333951 (owner: 10Krinkle)
[17:09:36] <wikibugs>	 (03CR) 10Krinkle: Add coverage report [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333863 (owner: 10Hashar)
[17:09:38] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] Add coverage report [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333863 (owner: 10Hashar)
[17:11:21] <wikibugs>	 (03Merged) 10jenkins-bot: Add coverage report [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333863 (owner: 10Hashar)
[17:16:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[17:38:00] <wikibugs>	 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12282448 (10VRiley-WMF) @fgiunchedi while trying to image the first server, I think it may be getting stuck on the RAID...
[17:48:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[17:52:12] <wikibugs>	 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12282514 (10VRiley-WMF) cloudvirt1072 D5 - U10 CableID: 5309 Port: 9  cloudvirt1073 D5 - U11 CableID: 5299 Port: 38  clo...
[17:53:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[17:54:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:09:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:12:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:16:04] <wikibugs>	 (03PS1) 10Krinkle: IrcClient: Faster RCReader startup by batching JOINs [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333987
[18:16:49] <wikibugs>	 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12282585 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1065....
[18:17:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[18:18:07] <wikibugs>	 (03CR) 10CI reject: [V:04-1] IrcClient: Faster RCReader startup by batching JOINs [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333987 (owner: 10Krinkle)
[18:22:33] <wikibugs>	 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Surprise harbor failure in wmcs.toolforge.k8s.prepare_upgrade - https://phabricator.wikimedia.org/T436126#12282627 (10BLiviero-WMF) Riccardo took a look at this and the auth info is coming from somewhere in this area: https://gerrit.wikimedia.o...
[18:26:44] <jinxer-wm>	 FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[18:28:38] <wikibugs>	 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Surprise harbor failure in wmcs.toolforge.k8s.prepare_upgrade - https://phabricator.wikimedia.org/T436126#12282644 (10Volans) The tests are run with: https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/wmcs-cookbooks/+/refs/heads/main/cookbook...
[18:30:22] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12282645 (10dancy) I went ahead and created an ext4 filesystem on /de...
[18:31:44] <jinxer-wm>	 RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors
[18:47:40] <wikibugs>	 10Tool-Pageviews, 07I18n: massviews-category-description lego for "category" - https://phabricator.wikimedia.org/T146973#12282703 (10MusikAnimal) 05In progress→03Resolved
[18:47:43] <wikibugs>	 10Tool-Pageviews: Add logarithmic URL query parameter - https://phabricator.wikimedia.org/T177369#12282705 (10MusikAnimal) 05In progress→03Resolved
[18:47:52] <wikibugs>	 10Tool-Pageviews: Add namespace filter to Massviews - https://phabricator.wikimedia.org/T166176#12282707 (10MusikAnimal) 05In progress→03Resolved
[18:48:17] <wikibugs>	 10Tool-Pageviews: Add import method to request massviews stats from multiple wiki's at once - https://phabricator.wikimedia.org/T204755#12282709 (10MusikAnimal) 05In progress→03Resolved
[18:48:25] <wikibugs>	 10Tool-Pageviews: [Topviews] Allow permalink to search results - https://phabricator.wikimedia.org/T193892#12282711 (10MusikAnimal) 05In progress→03Resolved
[18:48:35] <wikibugs>	 10Tool-Pageviews: Add option show moving average line over chart - https://phabricator.wikimedia.org/T147515#12282713 (10MusikAnimal) 05In progress→03Resolved
[18:48:46] <wikibugs>	 10Tool-Pageviews: Make permalink include chart properties - https://phabricator.wikimedia.org/T204620#12282715 (10MusikAnimal) 05In progress→03Stalled
[18:48:52] <wikibugs>	 10Tool-Pageviews: Make permalink include chart properties - https://phabricator.wikimedia.org/T204620#12282716 (10MusikAnimal) 05Stalled→03Resolved
[18:49:11] <wikibugs>	 10Tool-Pageviews, 06Data-Engineering, 06Data-Engineering-Icebox: Pageviews Analysis 2026 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#12282719 (10MusikAnimal) 05In progress→03Resolved
[19:10:46] <wikibugs>	 (03PS2) 10Krinkle: IrcClient: Faster RCReader startup by batching IRC joins [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333987
[19:13:36] <wikibugs>	 (03PS3) 10Krinkle: IrcClient: Faster RCReader startup by batching IRC joins [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333987
[19:17:30] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] IrcClient: Faster RCReader startup by batching IRC joins [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333987 (owner: 10Krinkle)
[19:18:52] <wikibugs>	 (03Merged) 10jenkins-bot: IrcClient: Faster RCReader startup by batching IRC joins [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1333987 (owner: 10Krinkle)
[19:54:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[19:59:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[20:03:06] <wikibugs>	 (03PS1) 10Gergő Tisza: frontend: Add new result format with one line per hit [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1334027
[20:04:45] <wikibugs>	 (03CR) 10Gergő Tisza: "Needed this for T436720, figured I would share it. Don't have strong feelings on whether it should be merged but I have occasionally wishe" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1334027 (owner: 10Gergő Tisza)
[20:07:27] <wikibugs>	 (03PS1) 10Hashar: build: drop coverage version binding [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334029
[20:09:09] <jinxer-wm>	 FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning
[20:13:52] <wikibugs>	 10Cloud-VPS (Project-requests), 10Tool-Pageviews: Request creation of pageviews VPS project - https://phabricator.wikimedia.org/T433305#12282993 (10MusikAnimal) 05Stalled→03Open Apologies for the long delay, though I'm not sure anyone was waiting on bated breath to finish this aside from me :-P  I've gone...
[20:14:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[20:17:00] <wikibugs>	 (03PS2) 10Krinkle: build: drop coverage version binding [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334029 (owner: 10Hashar)
[20:17:43] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] build: drop coverage version binding [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334029 (owner: 10Hashar)
[20:19:00] <wikibugs>	 (03Merged) 10jenkins-bot: build: drop coverage version binding [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334029 (owner: 10Hashar)
[20:21:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[20:21:49] <wikibugs>	 (03CR) 10Krinkle: [C:03+2] frontend: Add new result format with one line per hit [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1334027 (owner: 10Gergő Tisza)
[20:23:02] <wikibugs>	 (03Merged) 10jenkins-bot: frontend: Add new result format with one line per hit [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1334027 (owner: 10Gergő Tisza)
[20:31:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[20:41:59] <wikibugs>	 10Cloud-VPS, 06tools-infrastructure-team: Network unavailable on a few VMs - https://phabricator.wikimedia.org/T432426#12283049 (10Andrew) 05Open→03Resolved
[20:46:23] <wikibugs>	 (03PS1) 10Krinkle: ListManager: Tests for combining wl, al, and bot for the same user [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334044 (https://phabricator.wikimedia.org/T327129)
[20:54:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[20:59:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[21:00:53] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12283119 (10Southparkfan) >>! In T435393#12282645, @dancy wrote: > I...
[21:06:53] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12283145 (10Southparkfan) >>! In T429497#12282039, @Ottomata wrote: > @Southp...
[21:14:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[21:15:30] <wikibugs>	 10Tool-wikimedia-attribution, 06MediaWiki-API-Platform-Team, 10MediaWiki-REST-API: 'expand' property set as string instead of array in Attribution API - https://phabricator.wikimedia.org/T436865 (10HCoplin-WMF) 03NEW
[21:16:03] <wikibugs>	 10Tool-wikimedia-attribution, 06MediaWiki-API-Platform-Team, 10MediaWiki-REST-API: 'expand' property set as string instead of array in Attribution API - https://phabricator.wikimedia.org/T436865#12283202 (10HCoplin-WMF)
[21:19:38] <wmcs-alerts>	 FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[21:24:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[22:23:04] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace deployment-webperf2[12] (Bullseye deprecation) - https://phabricator.wikimedia.org/T436464#12283477 (10Southparkfan) 05In progress→03Resolved Done!
[22:24:31] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-docker-mathoid02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436465#12283480 (10Southparkfan) 05Open→03In progress a:03Southparkfan
[22:42:51] <wikibugs>	 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 07Epic, 06Release-Engineering-Team (Priority Backlog 📥): Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12283528 (10Southparkfan)
[23:08:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[23:09:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[23:14:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[23:17:04] <wmcs-alerts>	 FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[23:18:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[23:22:04] <wmcs-alerts>	 RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[23:22:59] <wmcs-alerts>	 FIRING: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[23:23:04] <wmcs-alerts>	 FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[23:23:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[23:27:38] <wmcs-alerts>	 FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[23:27:59] <wmcs-alerts>	 RESOLVED: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[23:28:04] <wmcs-alerts>	 RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[23:36:59] <wmcs-alerts>	 FIRING: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[23:37:03] <wmcs-alerts>	 FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[23:37:38] <wmcs-alerts>	 RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown
[23:41:59] <wmcs-alerts>	 RESOLVED: QuarryDown: Quarry application is unreachable   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown
[23:42:03] <wmcs-alerts>	 RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[23:48:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[23:50:33] <wikibugs>	 (03PS2) 10Krinkle: ListManager: Tests for combining wl, al, and bot for the same user [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334044 (https://phabricator.wikimedia.org/T327129)
[23:51:57] <wikibugs>	 (03PS1) 10Krinkle: ListManager: Fix adding to admin/bot list while on WL [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334149 (https://phabricator.wikimedia.org/T327129)
[23:53:42] <wikibugs>	 (03PS2) 10Krinkle: ListManager: Fix adding to admin/bot list while on WL [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334149 (https://phabricator.wikimedia.org/T327129)