[00:11:21] 10Striker, 06tools-platform-team: 500 internal error when accessing toolsadmin ssh-keys - https://phabricator.wikimedia.org/T434684#12249681 (10bd808) For future historians, the bug here was that once a running copy of #striker had created a new Developer account using `objectClass: wikimediaPerson` to sav... [01:34:18] (03open) 10samwilson: Add la-pt Transkribus model [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/15 (https://phabricator.wikimedia.org/T435843) [01:42:25] (03merge) 10samwilson: Add la-pt Transkribus model [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/15 (https://phabricator.wikimedia.org/T435843) [02:10:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:15:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:20:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:23:08] 10Toolforge, 06tools-infrastructure-team: [disable-tools] Disabled Toolforge tool deletions not occurring - https://phabricator.wikimedia.org/T431241#12249756 (10Andrew) >>! In T431241#12249229, @bd808 wrote: > The LDAP record looks like it has the expected attributes for maintain_kubeusers to see it as disabl... [02:25:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:10:50] (03update) 10zhaofjx: T435018: Wikinewsie footer elements misaligned on mobile view [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/7 (owner: 10ecoholic) [03:12:07] (03approved) 10zhaofjx: T435018: Wikinewsie footer elements misaligned on mobile view [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/7 (owner: 10ecoholic) [03:12:25] (03update) 10zhaofjx: T435018: Wikinewsie footer elements misaligned on mobile view [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/7 (owner: 10ecoholic) [03:13:20] (03merge) 10zhaofjx: T435018: Wikinewsie footer elements misaligned on mobile view [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/7 (owner: 10ecoholic) [03:21:54] 10Tool-wikinewsie: Wikinewsie footer elements misaligned on mobile view - https://phabricator.wikimedia.org/T435018#12249819 (10ZhaoFJx) >>! In T435018#12248893, @Akash_A wrote: > Have opened a PR, please do review and let me know if that does it. Thank you for your work! I have merged and deployed it. [03:22:57] 10Tool-wikinewsie: Wikinewsie footer elements misaligned on mobile view - https://phabricator.wikimedia.org/T435018#12249821 (10ZhaoFJx) 05Open→03Resolved [03:28:00] 10Tool-delintbot: Fix cases of tags not being closed correctly - https://phabricator.wikimedia.org/T417483#12249853 (10Redmin) 05In progress→03Open [03:52:50] (03merge) 10countcount: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/32 (owner: 10renovatebot) [03:52:56] (03merge) 10countcount: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/26 (owner: 10renovatebot) [03:53:06] (03merge) 10countcount: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/41 (owner: 10renovatebot) [03:53:22] (03merge) 10countcount: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/78 (owner: 10renovatebot) [04:21:22] 10Cloud-VPS (Debian Bullseye Deprecation), 10a Wikimedia CLI: Migrate mwcli away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435778#12249949 (10Samwilson) [04:39:39] (03update) 10bd808: Draft: Setup local and CI linters and tests [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/4 (https://phabricator.wikimedia.org/T435665) [04:40:04] (03update) 10bd808: Setup local and CI linters and tests [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/4 (https://phabricator.wikimedia.org/T435665) [04:42:10] (03update) 10bd808: Setup local and CI linters and tests [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/4 (https://phabricator.wikimedia.org/T435665) [04:56:08] (03merge) 10countcount: Update pnpm to v11.23.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/38 (owner: 10renovatebot) [04:56:20] (03merge) 10countcount: Update pnpm to v11.23.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/53 (owner: 10renovatebot) [04:56:24] (03merge) 10countcount: Update pnpm to v11.23.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/21 (owner: 10renovatebot) [04:56:30] (03merge) 10countcount: Update pnpm to v11.23.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/26 (owner: 10renovatebot) [04:56:35] (03merge) 10countcount: Update pnpm to v11.23.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/14 (owner: 10renovatebot) [04:56:58] (03merge) 10countcount: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/29 (owner: 10renovatebot) [05:14:03] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/13 [05:14:18] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/25 [05:14:32] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/20 [05:15:36] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/52 [05:16:30] (03update) 10renovatebot: Update cimg/rust Docker tag to v1.98 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/36 [05:17:14] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/37 [05:18:01] (03update) 10renovatebot: Update python Docker tag to v3.14.7 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [05:25:42] (03merge) 10countcount: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/37 (owner: 10renovatebot) [05:26:22] (03merge) 10countcount: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/52 (owner: 10renovatebot) [05:26:38] (03merge) 10countcount: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/20 (owner: 10renovatebot) [05:26:45] (03merge) 10countcount: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/25 (owner: 10renovatebot) [05:26:54] (03merge) 10countcount: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/13 (owner: 10renovatebot) [06:15:36] (03update) 10renovatebot: Update cimg/rust Docker tag to v1.98 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/36 [06:20:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:25:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:36:39] (03PS22) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [06:36:54] (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [07:07:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:12:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:17:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:17:49] 10Tool-inteGraality: Dashboard list/status page - https://phabricator.wikimedia.org/T435900 (10JeanFred) 03NEW [07:34:57] (03merge) 10dcaro: write_jobs_logs: rename from `jobs-api` to `jobs` source [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/33 (https://phabricator.wikimedia.org/T435798) [07:35:02] (03update) 10dcaro: refactor: made loki a package, and minor refactors [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/35 (https://phabricator.wikimedia.org/T432570) [07:35:07] !log taavi@cloudcumin1001 deployment-prep START - Cookbook wmcs.vps.remove_instance for instance deployment-ircd03 [07:35:30] !log taavi@cloudcumin1001 deployment-prep END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance deployment-ircd03 [07:36:24] (03merge) 10dcaro: api-gateway: change the logs-api url [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1387 (https://phabricator.wikimedia.org/T435798) [07:37:41] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.43-20260825073511-2dfe034d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1388 (https://phabricator.wikimedia.org/T435798) [07:40:56] 10Toolforge, 06tools-infrastructure-team: [disable-tools] Disabled Toolforge tool deletions not occurring - https://phabricator.wikimedia.org/T431241#12250199 (10taavi) ` Aug 11 10:46:05 tools-nfs-3 disable_tool.py[3411072]: INFO __main__: Tool vote-tracker is expired; archiving Aug 11 10:46:05 tools-nfs-3 dis... [07:41:25] FIRING: [2x] SystemdUnitFailed: libvirtd-tls.socket on cloudvirt1049:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:46:12] 10Toolforge, 06tools-platform-team: [builds-api] TOOL_DATA_DIR not available during `toolforge build start` execution - https://phabricator.wikimedia.org/T434369#12250213 (10DamianZaremba) > I think it would be great for both TOOL_DATA_DIR and the proposed TOOL_NAME variable to be available in both build and r... [07:51:25] FIRING: [3x] SystemdUnitFailed: libvirtd-tls.socket on cloudvirt1051:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:56:25] FIRING: [3x] SystemdUnitFailed: libvirtd-tls.socket on cloudvirt1051:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:00:07] 10Striker, 06tools-platform-team, 10Phabricator: Set corresponding project tag when creating a Diffusion repository via Striker - https://phabricator.wikimedia.org/T435906 (10Aklapper) 03NEW [08:00:39] 10Striker, 06tools-platform-team, 10Phabricator: Set corresponding project tag when creating a Diffusion repository via Striker - https://phabricator.wikimedia.org/T435906#12250242 (10Aklapper) p:05Triage→03Low [08:01:25] FIRING: [4x] SystemdUnitFailed: libvirtd-tls.socket on cloudvirt1051:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:03:59] 10Striker, 06tools-platform-team, 10Phabricator: Set corresponding project tag when creating a Diffusion repository via Striker - https://phabricator.wikimedia.org/T435906#12250249 (10Aklapper) [08:04:41] (03update) 10dcaro: write_job_logs: add jobs events [repos/cloud/toolforge/logs-api] (minor_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/34 (https://phabricator.wikimedia.org/T432570) [08:11:25] FIRING: [6x] SystemdUnitFailed: libvirtd-tls.socket on cloudvirt1049:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:11:36] (03CR) 10David Caro: [C:03+2] "I have been using this for a while now, merging." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1324758 (owner: 10David Caro) [08:12:56] (03Abandoned) 10David Caro: aptly: only maintain trixie support [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1322008 (https://phabricator.wikimedia.org/T434218) (owner: 10David Caro) [08:14:54] (03Merged) 10jenkins-bot: wmcs.toolforge.component.deploy: show tests already running [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1324758 (owner: 10David Caro) [08:15:15] (03approved) 10dcaro: logs-api: the url changes `jobs-api` to `jobs` [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/104 (https://phabricator.wikimedia.org/T435798) [08:15:20] (03merge) 10dcaro: logs-api: the url changes `jobs-api` to `jobs` [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/104 (https://phabricator.wikimedia.org/T435798) [08:17:17] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [08:17:40] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.105-20260825081532-ac46c749 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1389 (https://phabricator.wikimedia.org/T435798) [08:17:44] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.105-20260825081532-ac46c749 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1389 (https://phabricator.wikimedia.org/T435798) [08:21:25] FIRING: [5x] SystemdUnitFailed: libvirtd-tls.socket on cloudvirt1051:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:22:06] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [08:23:04] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [08:23:13] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 3 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12250304 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1003 for host cloudvirt... [08:26:55] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [08:27:27] (03approved) 10dcaro: api-gateway: bump to 0.0.105-20260825081532-ac46c749 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1389 (https://phabricator.wikimedia.org/T435798) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:27:33] (03merge) 10dcaro: api-gateway: bump to 0.0.105-20260825081532-ac46c749 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1389 (https://phabricator.wikimedia.org/T435798) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:27:34] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [08:30:20] (03update) 10dcaro: write_job_logs: add jobs events [repos/cloud/toolforge/logs-api] (minor_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/34 (https://phabricator.wikimedia.org/T432570) [08:31:06] (03update) 10dcaro: refactor: made loki a package, and minor refactors [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/35 (https://phabricator.wikimedia.org/T432570) [08:32:18] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [jobs-api] Send webservice start/stop/restart events to the new logs-api event log endpoint - https://phabricator.wikimedia.org/T432570#12250328 (10dcaro) a:03dcaro [08:35:33] (03update) 10dcaro: emailer: include pod status message in failure emails [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/58 (https://phabricator.wikimedia.org/T384252) (owner: 10mahveotm) [08:37:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [08:37:21] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [08:38:59] (03update) 10dcaro: models: support webservice job-type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) (owner: 10raymond-ndibe) [08:48:19] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [08:54:14] (03approved) 10dcaro: logs-api: bump to 0.0.43-20260825073511-2dfe034d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1388 (https://phabricator.wikimedia.org/T435798) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:54:19] (03update) 10dcaro: logs-api: bump to 0.0.43-20260825073511-2dfe034d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1388 (https://phabricator.wikimedia.org/T435798) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:54:32] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [08:54:45] !log dcaro@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component logs-api [08:54:53] (03merge) 10dcaro: logs-api: bump to 0.0.43-20260825073511-2dfe034d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1388 (https://phabricator.wikimedia.org/T435798) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:58:12] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-api] rename jobs-api source to jobs source for event logs - https://phabricator.wikimedia.org/T435798#12250428 (10dcaro) 05Open→03Resolved [09:11:48] FIRING: [2x] PuppetFailure: Puppet has failed on cloudvirt1054:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:36:39] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] send deployment events to the logs-api system-event log endpoint - https://phabricator.wikimedia.org/T432571#12250577 (10dcaro) I have a couple alternatives for how to handle these events. The main issue is filtering by job_name/component_name,... [09:41:48] FIRING: [4x] PuppetFailure: Puppet has failed on cloudvirt1054:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:45:39] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [jobs-api] Send webservice start/stop/restart events to the new logs-api event log endpoint - https://phabricator.wikimedia.org/T432570#12250606 (10dcaro) [09:46:18] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [jobs-api] Send webservice start/stop/restart events to the new logs-api event log endpoint - https://phabricator.wikimedia.org/T432570#12250607 (10dcaro) [09:55:20] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] Create endpoint to get tools based on a set config - https://phabricator.wikimedia.org/T433690#12250639 (10fnegri) [09:55:21] 10Toolforge, 06tools-platform-team: [components-api] Allow setting opt-in flags for special scraping control measures - https://phabricator.wikimedia.org/T433397#12250638 (10fnegri) [09:55:37] 10Toolforge, 06tools-platform-team, 07OKR-Work: Provide a way for users to enable and disable alerts - https://phabricator.wikimedia.org/T432864#12250640 (10fnegri) [09:55:38] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] Create endpoint to get tools based on a set config - https://phabricator.wikimedia.org/T433690#12250641 (10fnegri) [09:58:59] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12250657 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1003 for host cloudvirt1049... [10:05:39] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) [10:05:42] 10Tool-wheres-my-code-running: wheres-my-code-running thinks enwikinews is group1, misreports group1 version - https://phabricator.wikimedia.org/T435915 (10Lucas_Werkmeister_WMDE) 03NEW [10:30:18] RESOLVED: PuppetFailure: Puppet has failed on cloudvirt1057:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:30:54] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [10:30:57] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) [10:31:01] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [10:31:04] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) [10:31:09] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [10:31:11] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) [10:31:15] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [10:31:17] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) [10:31:20] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [10:31:23] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) [10:31:26] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance [10:31:29] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) [10:32:16] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12250780 (10fgiunchedi) 05In progress→03Resolved This is done, hosts have been relocated and are back in service [10:36:00] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#12250792 (10VRiley-WMF) 05Open→03Resolved This has been resolved [10:40:33] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#12250815 (10fgiunchedi) 05Resolved→03Open Unfortunately it hasn't, we still have a few rack mov... [10:40:46] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#12250818 (10fgiunchedi) a:05VRiley-WMF→03fgiunchedi [10:41:12] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#12250820 (10VRiley-WMF) I apologize, I thought that was all we had. [10:45:43] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921 (10fgiunchedi) 03NEW [10:46:44] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Physical move of servers - https://phabricator.wikimedia.org/T435922 (10VRiley-WMF) 03NEW [10:46:45] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923 (10fgiunchedi) 03NEW [10:46:54] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Physical move of servers - https://phabricator.wikimedia.org/T435922#12250901 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF This is completed [10:47:34] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#12250906 (10fgiunchedi) >>! In T424658#12250820, @VRiley-WMF wrote: > I apologize, I thought that w... [11:02:27] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] Create endpoint to get tools based on a set config - https://phabricator.wikimedia.org/T433690#12250990 (10dcaro) a:03dcaro [11:03:18] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] Create endpoint to get tools based on a set config - https://phabricator.wikimedia.org/T433690#12251007 (10dcaro) 05Open→03In progress [11:32:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:37:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:38:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:43:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:10:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:15:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:17:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:22:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:27:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:28:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:29:03] FIRING: ProjectProxyMainProxyInstanceDown: Proxy on proxy-5 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyInstanceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyInstanceDown [12:33:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:35:59] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [12:36:04] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [12:40:59] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [12:41:04] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [12:44:03] RESOLVED: ProjectProxyMainProxyInstanceDown: Proxy on proxy-5 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyInstanceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyInstanceDown [12:48:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:53:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:55:58] FIRING: MetricsinfraAlertmanagerDown: Metricsinfra alertmanager is unreachable #page - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MetricsinfraAlertmanagerDown - TODO - https://alerts.wikimedia.org/?q=alertname%3DMetricsinfraAlertmanagerDown [12:57:04] FIRING: ProjectProxyMainProxyInstanceDown: Proxy on proxy-5 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyInstanceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyInstanceDown [13:00:58] RESOLVED: MetricsinfraAlertmanagerDown: Metricsinfra alertmanager is unreachable #page - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MetricsinfraAlertmanagerDown - TODO - https://alerts.wikimedia.org/?q=alertname%3DMetricsinfraAlertmanagerDown [13:01:56] FIRING: HarborDown: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [13:03:50] 10Cloud-VPS (Debian Bullseye Deprecation), 10Internet In A Box: Iiab vps debian migration blocked on exposing rsync - https://phabricator.wikimedia.org/T433714#12251479 (10Tim-moody) 05Open→03Resolved a:03Tim-moody Migration is complete and the bullseye instance has been deleted. [13:06:56] RESOLVED: HarborDown: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [13:07:04] RESOLVED: ProjectProxyMainProxyInstanceDown: Proxy on proxy-5 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyInstanceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyInstanceDown [13:35:50] FIRING: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [13:36:03] FIRING: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:40:50] RESOLVED: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [13:41:03] RESOLVED: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:44:19] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#12251728 (10fgiunchedi) >>! In T403154#12242486, @BTullis wrote: > I believe that @bking will be able to ru... [13:45:01] 10PAWS, 06tools-infrastructure-team, 06tools-platform-team: Make sure dumps-nfs mounts are propagated inside PAWS containers - https://phabricator.wikimedia.org/T432583#12251751 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi failover complete [13:54:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:55:06] 10Data-Services, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-07 - 2026-08-28): Remove hdfs fuse mounts from clouddumps - https://phabricator.wikimedia.org/T434794#12251801 (10taavi) >>! In T434794#12237421, @BTullis wrote: > With regard to this: >> Actually, the mounts even seem to be compl... [13:59:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:04:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:09:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:21:20] (03open) 10dcaro: global: add public tool search with config endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/186 (https://phabricator.wikimedia.org/T433690) [14:24:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:24:39] 10VPS-project-Wikistats: Wikistats for wikipedias is not reachable - https://phabricator.wikimedia.org/T435959 (10Xqt) 03NEW [14:26:01] 10VPS-project-Wikistats: Wikistats for wikipedias is not reachable - https://phabricator.wikimedia.org/T435959#12252011 (10RhinosF1) p:05Triage→03Unbreak! a:03RhinosF1 [14:32:25] 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance: [components-api] do not expose rebuild_storage at the api level - https://phabricator.wikimedia.org/T434780#12252083 (10dcaro) a:03dcaro [14:32:28] 10Toolforge, 06tools-platform-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance: [components-api] do not expose rebuild_storage at the api level - https://phabricator.wikimedia.org/T434780#12252085 (10dcaro) 05Open→03In progress [14:33:26] (03open) 10dcaro: openapi: remove `rebuild_storage` from the interface [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/187 (https://phabricator.wikimedia.org/T434780) [14:34:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:34:46] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/32859995726 (https://github.com/cluebotng/component-configs/commits/8e54667ba485459bad6bd7798274c98ea487caee) [14:34:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [14:35:41] (03update) 10dcaro: openapi: remove `rebuild_storage` from the interface [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/187 (https://phabricator.wikimedia.org/T434780) [14:35:45] 10VPS-project-Wikistats: Wikistats for wikipedias is not reachable - https://phabricator.wikimedia.org/T435959#12252138 (10Xqt) Two other clusters are also down: - https://wikistats.wmcloud.org/display.php?t=et - https://wikistats.wmcloud.org/display.php?t=si [14:37:08] (03update) 10dcaro: global: add public tool search with config endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/186 (https://phabricator.wikimedia.org/T433690) [14:37:31] (03update) 10dcaro: global: add public tool search with config endpoint [repos/cloud/toolforge/components-api] (avoid_rebuild_storage_showing) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/186 (https://phabricator.wikimedia.org/T433690) [14:38:25] 10VPS-project-Wikistats: Wikistats for wikipedias is not reachable - https://phabricator.wikimedia.org/T435959#12252172 (10RhinosF1) I'm very confused, the logs are full of undefined array keys but this started just a few hours ago. The code hasn't changed in months. [14:40:15] 10VPS-project-Wikistats: Wikistats for wikipedias is not reachable - https://phabricator.wikimedia.org/T435959#12252180 (10RhinosF1) I restarted apache2 and it fixed it but it doesn't make much sense as to why it was failing [14:40:31] 10VPS-project-Wikistats: Wikistats for wikipedias is not reachable - https://phabricator.wikimedia.org/T435959#12252182 (10RhinosF1) p:05Unbreak!→03Medium a:05RhinosF1→03None [14:44:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:46:01] 10VPS-project-Wikistats: Wikistats for wikipedias is not reachable - https://phabricator.wikimedia.org/T435959#12252228 (10Xqt) >>! In T435959#12252172, @RhinosF1 wrote: > I'm very confused, the logs are full of undefined array keys but this started just a few hours ago. The code hasn't changed in months. As I... [14:46:09] 10Tool-wikinewsie: Deaths story - https://phabricator.wikimedia.org/T435844#12252231 (10Pharos) I guess the easiest way to do this might be to exclude this category: https://en.wikipedia.org/wiki/Category:Lists_of_deaths_in_2026 [14:46:28] 10VPS-project-Wikistats: Wikistats for wikipedias is not reachable - https://phabricator.wikimedia.org/T435959#12252233 (10RhinosF1) ` root@wikistats-bookworm:/var/log/apache2# df -h Filesystem Size Used Avail Use% Mounted on udev 2.0G 0 2.0G 0% /dev tmpfs 393M 588K 393M 1%... [14:47:23] 10Cloud-VPS (Quota-requests): Quota increase request for project wikispeech - https://phabricator.wikimedia.org/T435967 (10Sebastian_Berlin-WMSE) 03NEW [14:47:55] 10Cloud-VPS (Quota-requests): Quota increase request for project wikispeech - https://phabricator.wikimedia.org/T435967#12252248 (10Sebastian_Berlin-WMSE) [14:48:20] (03update) 10dcaro: global: add public tool search with config endpoint [repos/cloud/toolforge/components-api] (avoid_rebuild_storage_showing) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/186 (https://phabricator.wikimedia.org/T433690) [14:54:04] 10VPS-project-Wikistats: Wikistats for wikipedias is not reachable - https://phabricator.wikimedia.org/T435959#12252285 (10RhinosF1) I cleaned some space up but I'll chat to @Dzahn about further investgating [14:59:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:14:41] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/15 [15:14:44] (03open) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/15 [15:15:02] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/27 [15:15:07] (03open) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/27 [15:15:21] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/22 [15:15:26] (03open) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/22 [15:16:27] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/54 [15:16:31] (03open) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/54 [15:17:33] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/39 [15:17:36] 10Tool-inteGraality: Queries endpoint support for Property columns with qualifier - https://phabricator.wikimedia.org/T251008#12252435 (10JeanFred) 05Open→03Resolved a:03JeanFred [15:17:37] (03open) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/39 [15:24:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:29:28] 10Tool-inteGraality: Dashboard list/status page - https://phabricator.wikimedia.org/T435900#12252541 (10JeanFred) [15:29:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:30:53] (03update) 10dcaro: global: add public tool search with config endpoint [repos/cloud/toolforge/components-api] (avoid_rebuild_storage_showing) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/186 (https://phabricator.wikimedia.org/T433690) [15:34:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:37:21] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/32866638493 (https://github.com/cluebotng/component-configs/commits/f0c0373d1612550e858604e6c40555d78d071318) [15:37:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [15:45:33] 10Toolforge, 06tools-platform-team, 07OKR-Work: Add scaffolding for the Toolforge Alerting Service - https://phabricator.wikimedia.org/T435971 (10fnegri) 03NEW [15:45:40] 10Tool-paulina: Incorrect URL resolution for Wikidata entities - https://phabricator.wikimedia.org/T435972 (10Danielyepezgarces) 03NEW [15:46:48] 10Tool-paulina: Incorrect URL resolution for Wikidata entities - https://phabricator.wikimedia.org/T435972#12252642 (10Danielyepezgarces) [15:50:57] (03open) 10dancy: Update group1 representative wiki [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/5 (https://phabricator.wikimedia.org/T421796 https://phabricator.wikimedia.org/T435915) [15:53:37] (03approved) 10bd808: Update group1 representative wiki [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/5 (https://phabricator.wikimedia.org/T421796 https://phabricator.wikimedia.org/T435915) (owner: 10dancy) [15:57:01] 10Toolforge, 06tools-platform-team, 07OKR-Work: Email tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865#12252684 (10fnegri) > we can use in-memory for starters until we need to scale up @dcaro your proposal of using polling without a persistent storage is inter... [15:58:10] (03merge) 10dancy: Update group1 representative wiki [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/5 (https://phabricator.wikimedia.org/T421796 https://phabricator.wikimedia.org/T435915) [15:58:31] (03merge) 10dancy: Setup local and CI linters and tests [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/4 (https://phabricator.wikimedia.org/T435665) (owner: 10bd808) [16:16:18] 10Toolforge, 06tools-platform-team: [toolsdb] 2026-08-18 Transaction History Length growing too much - https://phabricator.wikimedia.org/T435220#12252808 (10fnegri) Current growth rate: {F100058046} [16:19:23] (03open) 10bd808: Fix Procfile to work with refactored code [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/6 [16:19:48] (03approved) 10dancy: Fix Procfile to work with refactored code [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/6 (owner: 10bd808) [16:20:48] (03merge) 10bd808: Fix Procfile to work with refactored code [toolforge-repos/wheres-my-code-running] - 10https://gitlab.wikimedia.org/toolforge-repos/wheres-my-code-running/-/merge_requests/6 [16:25:28] 10Tool-wheres-my-code-running, 13Patch-For-Review: wheres-my-code-running thinks enwikinews is group1, misreports group1 version - https://phabricator.wikimedia.org/T435915#12252852 (10dancy) 05Open→03Resolved a:03dancy Fixed. [16:32:38] 10Cloud-VPS, 06tools-infrastructure-team: Fold special maps proxies back to the normal proxy - https://phabricator.wikimedia.org/T431284#12252904 (10Andrew) This sounds great to me, although maybe we want a little bit of settling in with the new dynamic proxy setup first? [16:36:04] 10Toolforge, 06tools-platform-team, 07OKR-Work: Add scaffolding for the Toolforge Alerting Service - https://phabricator.wikimedia.org/T435971#12252909 (10fnegri) p:05Triage→03High [16:36:14] 10Toolforge, 06tools-platform-team: [toolsdb] 2026-08-18 Transaction History Length growing too much - https://phabricator.wikimedia.org/T435220#12252911 (10magnusmanske) I have fixed some queries using the wrong FORCE INDEX, that should help. I am backfilling some backlog tasks, the current load is not indici... [16:38:17] 10Toolforge, 06tools-platform-team: [toolsdb] 2026-08-18 Transaction History Length growing too much - https://phabricator.wikimedia.org/T435220#12252940 (10fnegri) 05In progress→03Resolved > I am backfilling some backlog tasks, the current load is not indicitative of long-term behaviour. Thanks, that... [16:41:17] 10Tool-paulina: Incorrect URL resolution for Wikidata entities - https://phabricator.wikimedia.org/T435972#12252976 (10Pepe_piton) @Danielyepezgarces Good suggestion, thanks! I implemented it with https://gitlab.wikimedia.org/toolforge-repos/paulina/-/commit/d9d52342460f24356380d2297fa741307d34afad Basicall... [16:41:30] 10Tool-paulina: Incorrect URL resolution for Wikidata entities - https://phabricator.wikimedia.org/T435972#12252977 (10Pepe_piton) 05Open→03Resolved a:03Pepe_piton [16:59:05] 10Toolforge, 06tools-platform-team, 07OKR-Work: Email tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865#12253066 (10dcaro) > Let me know if there are objections to this plan. In the meantime, I will start creating the scaffolding of the Toolforge Alerting Servic... [17:26:53] 10Toolforge, 06tools-platform-team, 07OKR-Work: Email tool maintainers when an alert condition is detected - https://phabricator.wikimedia.org/T432865#12253216 (10taavi) >>! In T432865#12252684, @fnegri wrote: > Please note the PoC is polling the Prometheus API directly instead of the Alertmanager API: the i... [17:41:44] 10Tool-paulina: Incorrect URL resolution for Wikidata entities - https://phabricator.wikimedia.org/T435972#12253282 (10Danielyepezgarces) For now, I think the change is good :) [18:49:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:54:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:14:54] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/15 [19:15:10] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/27 [19:15:25] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/22 [19:16:27] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/54 [19:17:27] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/39 [19:18:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:23:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:14:20] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/15 [20:14:27] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/27 [20:14:34] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/22 [20:15:29] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/54 [20:16:10] (03update) 10renovatebot: Update pnpm to v11.24.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/39 [20:27:56] 10Tool-wikinewsie: Multimedia focus original reporting on Wikinewsie - https://phabricator.wikimedia.org/T435264#12254132 (10Pharos) I was thinking, maybe we should just call this the "Snapshot" feature, as a multimedia+caption element that's more personalized / original to an individual trusted Wikimedia contri... [20:42:03] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/labs/private on instance paws-puppetserver-1 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [20:47:01] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Doing 😎): Planning for a plan to migrate Beta Cluster again - https://phabricator.wikimedia.org/T433998#12254200 (10Southparkfan) @bd808 and I have The plan still needs work before it can be shared and used.... [21:18:03] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-opensearch-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:38:03] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance tools-opensearch-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:39:33] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance tools-opensearch-2 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:44:33] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance tools-opensearch-1 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:52:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:54:33] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance tools-opensearch-1 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:04:33] RESOLVED: [2x] PuppetAgentNoResources: No Puppet resources found on instance tools-opensearch-1 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:07:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:30:22] (03open) 10waldir: Fix over-cropping of images [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/8 [23:37:07] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 13Patch-For-Review: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12254725 (10bd808) When I [[https://gitlab.wiki... [23:38:19] (03update) 10waldir: Fix over-cropping of images [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/8 [23:38:31] (03update) 10waldir: Fix over-cropping of images [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/8 [23:38:38] (03update) 10waldir: Fix over-cropping of images [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/8 [23:41:38] 10Tool-wikinewsie: Wikinewsie images often off-center - https://phabricator.wikimedia.org/T431881#12254738 (10waldyrious) I've submitted https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/8 which provides an alternative solution to this: instead of moving the image hoping that the cropp... [23:41:54] 10Tool-wikinewsie: Adjustable image alignment on Wikinewsie - https://phabricator.wikimedia.org/T434467#12254740 (10waldyrious) I've submitted https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/8 which provides an alternative solution to this: instead of moving the image hoping that the cro... [23:48:32] 10Tool-wikinewsie: Dynamically adapt to the OS theme preference (prefers-color-scheme) - https://phabricator.wikimedia.org/T436041 (10waldyrious) 03NEW [23:49:10] 10Tool-wikinewsie: Dynamically adapt to the OS theme preference (prefers-color-scheme) - https://phabricator.wikimedia.org/T436041#12254796 (10waldyrious) Implemented in https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/5 (I'm creating this task retroactively, for tracking purposes).