[00:05:38] (03update) 10mahveotm: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) [01:14:26] (03PS1) 10Krinkle: IrcClient: Increase timeout from 200s to 250s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1336253 [01:16:29] (03PS2) 10Krinkle: IrcClient: Increase timeout from 200s to 250s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1336253 [01:22:58] (03PS3) 10Krinkle: IrcClient: Increase timeout from 200s to 250s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1336253 [01:23:01] (03CR) 10Krinkle: [C:03+2] IrcClient: Increase timeout from 200s to 250s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1336253 (owner: 10Krinkle) [01:24:26] (03Merged) 10jenkins-bot: IrcClient: Increase timeout from 200s to 250s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1336253 (owner: 10Krinkle) [01:44:57] (03open) 10legoktm: Switch to rocket_tera [toolforge-repos/cativity] - 10https://gitlab.wikimedia.org/toolforge-repos/cativity/-/merge_requests/5 [01:45:07] (03open) 10legoktm: Switch to rocket_tera [toolforge-repos/ldap] - 10https://gitlab.wikimedia.org/toolforge-repos/ldap/-/merge_requests/8 [01:45:13] (03open) 10legoktm: Switch to rocket_tera [toolforge-repos/cs1-fixer] - 10https://gitlab.wikimedia.org/toolforge-repos/cs1-fixer/-/merge_requests/4 [01:53:24] (03update) 10legoktm: Switch to rocket_tera [toolforge-repos/ldap] - 10https://gitlab.wikimedia.org/toolforge-repos/ldap/-/merge_requests/8 [02:08:24] (03merge) 10legoktm: Switch to rocket_tera [toolforge-repos/cativity] - 10https://gitlab.wikimedia.org/toolforge-repos/cativity/-/merge_requests/5 [02:11:01] (03merge) 10legoktm: Switch to rocket_tera [toolforge-repos/cs1-fixer] - 10https://gitlab.wikimedia.org/toolforge-repos/cs1-fixer/-/merge_requests/4 [02:11:27] (03merge) 10legoktm: Switch to rocket_tera [toolforge-repos/ldap] - 10https://gitlab.wikimedia.org/toolforge-repos/ldap/-/merge_requests/8 [02:15:07] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [02:15:41] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [02:15:42] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [02:16:03] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [02:16:47] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24 [02:19:23] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [02:22:59] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [02:23:10] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [02:23:32] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [02:23:41] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [02:23:53] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [02:24:01] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [02:24:10] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [02:24:20] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [02:57:20] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/adminscore] - 10https://gitlab.wikimedia.org/toolforge-repos/adminscore/-/merge_requests/1 [02:58:37] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/apt-browser] - 10https://gitlab.wikimedia.org/toolforge-repos/apt-browser/-/merge_requests/9 [02:58:50] 10Tool-wikinewsie: Adjustable image alignment on Wikinewsie - https://phabricator.wikimedia.org/T434467#12291982 (10Pharos) >>! In T434467#12291550, @waldyrious wrote: > It seems to me that the main issue is with images that have a transparent background, right? It's the majority of the current really weird ima... [02:59:23] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/cs1-fixer] - 10https://gitlab.wikimedia.org/toolforge-repos/cs1-fixer/-/merge_requests/5 [03:00:49] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/cativity] - 10https://gitlab.wikimedia.org/toolforge-repos/cativity/-/merge_requests/6 [03:01:35] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/empty-categories] - 10https://gitlab.wikimedia.org/toolforge-repos/empty-categories/-/merge_requests/7 [03:02:28] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/echo-chamber] - 10https://gitlab.wikimedia.org/toolforge-repos/echo-chamber/-/merge_requests/1 [03:06:40] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/ldap] - 10https://gitlab.wikimedia.org/toolforge-repos/ldap/-/merge_requests/9 [03:07:29] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/fast-ec] - 10https://gitlab.wikimedia.org/toolforge-repos/fast-ec/-/merge_requests/3 [03:08:31] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/newusers] - 10https://gitlab.wikimedia.org/toolforge-repos/newusers/-/merge_requests/6 [03:16:12] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/adminscore] - 10https://gitlab.wikimedia.org/toolforge-repos/adminscore/-/merge_requests/1 [03:16:34] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [03:16:56] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [03:18:20] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [03:20:13] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/newusers] - 10https://gitlab.wikimedia.org/toolforge-repos/newusers/-/merge_requests/6 [03:20:18] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [03:20:26] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [03:20:33] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [03:21:14] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/apt-browser] - 10https://gitlab.wikimedia.org/toolforge-repos/apt-browser/-/merge_requests/9 [03:22:37] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/echo-chamber] - 10https://gitlab.wikimedia.org/toolforge-repos/echo-chamber/-/merge_requests/1 [03:28:04] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/cativity] - 10https://gitlab.wikimedia.org/toolforge-repos/cativity/-/merge_requests/6 [03:33:26] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/empty-categories] - 10https://gitlab.wikimedia.org/toolforge-repos/empty-categories/-/merge_requests/7 [03:39:40] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/cs1-fixer] - 10https://gitlab.wikimedia.org/toolforge-repos/cs1-fixer/-/merge_requests/5 [03:53:55] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/ldap] - 10https://gitlab.wikimedia.org/toolforge-repos/ldap/-/merge_requests/9 [03:55:00] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/fast-ec] - 10https://gitlab.wikimedia.org/toolforge-repos/fast-ec/-/merge_requests/3 [04:24:34] 10Quarry, 06tools-platform-team: [bug] - https://phabricator.wikimedia.org/T437125 (10BrianYoung1) 03NEW [05:23:25] 10Tool-wikimonitor, 07patch-welcome: Saved edit summaries for faster undo and rollback actions - https://phabricator.wikimedia.org/T428349#12292052 (10CodeSparks45) a:03CodeSparks45 [05:46:41] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Arusha 30): Apply the 4x-iops Cinder volume type to the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T437127 (10Mhurd) 03NEW [05:52:04] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Arusha 30): Apply the 4x-iops Cinder volume type to the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T437127#12292073 (10Mhurd) [05:52:05] 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Arusha 30): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12292074 (10Mhurd) [06:09:31] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Arusha 30): Apply the 4x-iops Cinder volume type to the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T437127#12292075 (10Mhurd) [06:11:15] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Arusha 30): Apply the 4x-iops Cinder volume type to the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T437127#12292076 (10Mhurd) [06:14:13] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Arusha 30): Apply the 4x-iops Cinder volume type to the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T437127#12292077 (10Mhurd) [06:17:45] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Arusha 30): Apply the 4x-iops Cinder volume type to the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T437127#12292078 (10Mhurd) [07:14:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:19:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:33:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:38:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:57:29] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, 10wikimedia-irc-libera: Promote use of SASL for Cloud VPS/Toolforge hosted Libera.chat / Freenode IRC bots - https://phabricator.wikimedia.org/T278584#12292140 (10Krinkle) CVNBot now implements SASL and uses it for Libera Chat... [08:10:54] (03PS1) 10Krinkle: IrcClient: Increase timeout from 250s to 600s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1336585 [08:11:06] (03CR) 10Krinkle: [C:03+2] IrcClient: Increase timeout from 250s to 600s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1336585 (owner: 10Krinkle) [08:12:24] (03Merged) 10jenkins-bot: IrcClient: Increase timeout from 250s to 600s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1336585 (owner: 10Krinkle) [08:50:46] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, 10wikimedia-irc-libera: Promote use of SASL for Cloud VPS/Toolforge hosted Libera.chat / Freenode IRC bots - https://phabricator.wikimedia.org/T278584#12292170 (10Krinkle) [09:21:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:22:00] (03PS1) 10Krinkle: write_config: Add mediawiki/vendor to "Everything" preset [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1336608 (https://phabricator.wikimedia.org/T227704) [09:22:23] 10VPS-project-Codesearch, 06MediaWiki-Core-Platform-Team (Radar), 13Patch-For-Review: Codesearch: "mediawiki/vendor" is missing from Everything preset - https://phabricator.wikimedia.org/T227704#12292219 (10Krinkle) p:05Triage→03Medium [09:22:34] 10VPS-project-Codesearch, 06MediaWiki-Core-Platform-Team (Radar), 13Patch-For-Review: Codesearch: "mediawiki/vendor" is missing from Everything preset - https://phabricator.wikimedia.org/T227704#12292220 (10Krinkle) [09:26:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:47:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:52:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:59:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:09:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:33:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:38:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:58:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:03:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:42:08] 10Tool-linkdata: link_attribution is 0.33 on enwiki: 67% of blessed inlinks have no provenance match - https://phabricator.wikimedia.org/T437138 (10Audiodude) 03NEW p:05Triage→03Medium [13:16:35] 10Tool-linkdata: link_attribution is 0.33 on enwiki: 67% of blessed inlinks have no provenance match - https://phabricator.wikimedia.org/T437138#12292429 (10Audiodude) 05Open→03Resolved a:03Audiodude Investigated by resampling from the replica (staging was already pruned after publish): 400 random ns-0... [14:17:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [14:20:52] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/otd-helper] - 10https://gitlab.wikimedia.org/toolforge-repos/otd-helper/-/merge_requests/6 [14:21:14] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/replag-embed] - 10https://gitlab.wikimedia.org/toolforge-repos/replag-embed/-/merge_requests/4 [14:21:37] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/poty-stuff] - 10https://gitlab.wikimedia.org/toolforge-repos/poty-stuff/-/merge_requests/5 [14:21:42] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/prosesize] - 10https://gitlab.wikimedia.org/toolforge-repos/prosesize/-/merge_requests/6 [14:22:13] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/rust-hello-world] - 10https://gitlab.wikimedia.org/toolforge-repos/rust-hello-world/-/merge_requests/8 [14:22:20] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/streaks] - 10https://gitlab.wikimedia.org/toolforge-repos/streaks/-/merge_requests/7 [14:22:52] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/upcoming-mainpage] - 10https://gitlab.wikimedia.org/toolforge-repos/upcoming-mainpage/-/merge_requests/7 [14:22:55] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/tour-nyc] - 10https://gitlab.wikimedia.org/toolforge-repos/tour-nyc/-/merge_requests/3 [14:23:14] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/supertranscount] - 10https://gitlab.wikimedia.org/toolforge-repos/supertranscount/-/merge_requests/2 [14:43:03] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12292493 (10Andrew) Just got a stalled read on 1046. So the scheduler change did not resolve this. [14:43:59] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/streaks] - 10https://gitlab.wikimedia.org/toolforge-repos/streaks/-/merge_requests/7 [14:44:06] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/tour-nyc] - 10https://gitlab.wikimedia.org/toolforge-repos/tour-nyc/-/merge_requests/3 [14:47:39] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/poty-stuff] - 10https://gitlab.wikimedia.org/toolforge-repos/poty-stuff/-/merge_requests/5 [14:51:31] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/rust-hello-world] - 10https://gitlab.wikimedia.org/toolforge-repos/rust-hello-world/-/merge_requests/8 [14:52:20] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/otd-helper] - 10https://gitlab.wikimedia.org/toolforge-repos/otd-helper/-/merge_requests/6 [14:53:50] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/upcoming-mainpage] - 10https://gitlab.wikimedia.org/toolforge-repos/upcoming-mainpage/-/merge_requests/7 [14:57:03] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/supertranscount] - 10https://gitlab.wikimedia.org/toolforge-repos/supertranscount/-/merge_requests/2 [15:05:40] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/replag-embed] - 10https://gitlab.wikimedia.org/toolforge-repos/replag-embed/-/merge_requests/4 [15:10:51] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/prosesize] - 10https://gitlab.wikimedia.org/toolforge-repos/prosesize/-/merge_requests/6 [15:41:10] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/shorturls] - 10https://gitlab.wikimedia.org/toolforge-repos/shorturls/-/merge_requests/4 [15:41:15] (03open) 10legoktm: Updates (Sept. 2026) [toolforge-repos/rfa-voting-history] - 10https://gitlab.wikimedia.org/toolforge-repos/rfa-voting-history/-/merge_requests/4 [15:52:36] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/shorturls] - 10https://gitlab.wikimedia.org/toolforge-repos/shorturls/-/merge_requests/4 [15:54:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:59:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:03:34] (03merge) 10legoktm: Updates (Sept. 2026) [toolforge-repos/rfa-voting-history] - 10https://gitlab.wikimedia.org/toolforge-repos/rfa-voting-history/-/merge_requests/4 [16:03:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:08:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:57:28] FIRING: NodeTextfileStale: Stale textfile for clouddumps1002:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [18:17:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:22:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:41:13] 10Quarry, 06tools-platform-team, 06DBA: access to database table linktarget is slow - https://phabricator.wikimedia.org/T437149 (10Herzi.Pinki) 03NEW [18:46:41] 10Quarry, 06tools-platform-team, 06DBA: access to database table linktarget is slow - https://phabricator.wikimedia.org/T437149#12292664 (10taavi) 05Open→03Declined Your query is missing a namespace filter which is required to efficiently query the table by title. [18:57:09] 10Data-Services, 06tools-platform-team, 06DBA: access to database table linktarget is slow - https://phabricator.wikimedia.org/T437149#12292673 (10JJMC89) [19:01:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:06:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:19:57] (03open) 10legoktm: Switch to rocket_tera [toolforge-repos/apt-browser] - 10https://gitlab.wikimedia.org/toolforge-repos/apt-browser/-/merge_requests/10 [20:21:24] (03open) 10legoktm: Switch to rocket_tera [toolforge-repos/tour-nyc] - 10https://gitlab.wikimedia.org/toolforge-repos/tour-nyc/-/merge_requests/4 [20:22:32] (03open) 10legoktm: Switch to rocket_tera [toolforge-repos/echo-chamber] - 10https://gitlab.wikimedia.org/toolforge-repos/echo-chamber/-/merge_requests/2 [20:23:25] (03open) 10legoktm: Switch to rocket_tera [toolforge-repos/prosesize] - 10https://gitlab.wikimedia.org/toolforge-repos/prosesize/-/merge_requests/7 [20:24:32] (03open) 10legoktm: Switch to rocket_tera [toolforge-repos/supertranscount] - 10https://gitlab.wikimedia.org/toolforge-repos/supertranscount/-/merge_requests/3 [20:25:16] (03open) 10legoktm: Switch to rocket_tera [toolforge-repos/empty-categories] - 10https://gitlab.wikimedia.org/toolforge-repos/empty-categories/-/merge_requests/8 [20:37:48] (03merge) 10legoktm: Switch to rocket_tera [toolforge-repos/tour-nyc] - 10https://gitlab.wikimedia.org/toolforge-repos/tour-nyc/-/merge_requests/4 [20:38:17] (03merge) 10legoktm: Switch to rocket_tera [toolforge-repos/apt-browser] - 10https://gitlab.wikimedia.org/toolforge-repos/apt-browser/-/merge_requests/10 [20:39:15] (03merge) 10legoktm: Switch to rocket_tera [toolforge-repos/empty-categories] - 10https://gitlab.wikimedia.org/toolforge-repos/empty-categories/-/merge_requests/8 [20:50:16] (03merge) 10legoktm: Switch to rocket_tera [toolforge-repos/echo-chamber] - 10https://gitlab.wikimedia.org/toolforge-repos/echo-chamber/-/merge_requests/2 [20:50:28] (03merge) 10legoktm: Switch to rocket_tera [toolforge-repos/supertranscount] - 10https://gitlab.wikimedia.org/toolforge-repos/supertranscount/-/merge_requests/3 [20:56:27] (03merge) 10legoktm: Switch to rocket_tera [toolforge-repos/prosesize] - 10https://gitlab.wikimedia.org/toolforge-repos/prosesize/-/merge_requests/7 [21:57:43] FIRING: NodeTextfileStale: Stale textfile for clouddumps1002:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [21:59:57] FIRING: ProbeDown: Service labweb-ssl:7443 has failed probes (http_labweb-ssl_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#labweb-ssl:7443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:04:57] RESOLVED: ProbeDown: Service labweb-ssl:7443 has failed probes (http_labweb-ssl_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#labweb-ssl:7443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [23:50:54] (03open) 10tmoney: Replace batch paste form with the mw-selections picker [toolforge-repos/findlinkfast] - 10https://gitlab.wikimedia.org/toolforge-repos/findlinkfast/-/merge_requests/1 [23:52:40] (03update) 10tmoney: Replace batch paste form with the mw-selections picker [toolforge-repos/findlinkfast] - 10https://gitlab.wikimedia.org/toolforge-repos/findlinkfast/-/merge_requests/1 [23:53:12] (03merge) 10tmoney: Replace batch paste form with the mw-selections picker [toolforge-repos/findlinkfast] - 10https://gitlab.wikimedia.org/toolforge-repos/findlinkfast/-/merge_requests/1