[00:03:04] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [00:08:04] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [00:16:08] 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12283790 (10Ottomata) 🫡 Already added to task description and noted to Thoma... [01:04:49] (03PS1) 10Krinkle: [WIP] Release 5.0.0 [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334182 [01:14:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [01:24:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [01:25:07] (03update) 10samwilson: Upgrade to Symfony 8.1 [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/12 (https://phabricator.wikimedia.org/T432808) [01:34:12] (03PS2) 10Krinkle: [WIP] Release 5.0.0 [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334182 [02:11:56] FIRING: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:16:56] RESOLVED: ProbeDown: Service tools-k8s-haproxy-8:443 has failed probes (http_admin_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:23:06] (03PS3) 10Krinkle: Release 5.0.0 [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334182 [02:23:54] (03PS4) 10Krinkle: Release 5.0.0 [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334182 [02:31:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:36:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:03:34] 10Tool-wsindex, 10Wikisource Reader App (Android): Add descriptions for books where Wikipedia articles are available [API] - https://phabricator.wikimedia.org/T430400#12283960 (10Bodhisattwa) [03:03:41] 10Tool-wsindex, 10Wikisource Reader App (Android): Add descriptions for books where Wikipedia articles are available [API] - https://phabricator.wikimedia.org/T430400#12283963 (10Bodhisattwa) [03:10:01] 10Tool-wsindex, 10Wikisource Reader App (Android): Add summary statistics to analytics page - https://phabricator.wikimedia.org/T430399#12283971 (10Bodhisattwa) [03:34:20] (03PS5) 10Krinkle: Release 5.0.0 [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334182 [03:35:09] (03PS6) 10Krinkle: Release 5.0.0 [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334182 [03:40:22] (03PS7) 10Krinkle: Release 5.0.0 [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334182 [03:42:07] (03update) 10raymond-ndibe: logs-api: bump to 0.0.47-20260902135501-fabf4256 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1396 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:42:09] (03approved) 10raymond-ndibe: logs-api: bump to 0.0.47-20260902135501-fabf4256 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1396 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:42:13] (03merge) 10raymond-ndibe: logs-api: bump to 0.0.47-20260902135501-fabf4256 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1396 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:42:22] (03update) 10raymond-ndibe: calico: bump to 0.0.20-20260902135340-31d4b17e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1399 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:42:28] (03approved) 10raymond-ndibe: calico: bump to 0.0.20-20260902135340-31d4b17e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1399 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:42:31] (03update) 10raymond-ndibe: calico: bump to 0.0.20-20260902135340-31d4b17e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1399 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:42:57] (03merge) 10raymond-ndibe: calico: bump to 0.0.20-20260902135340-31d4b17e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1399 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:43:08] (03update) 10raymond-ndibe: components-api: bump to 0.0.212-20260902135855-3e34d704 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1401 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:43:09] (03approved) 10raymond-ndibe: components-api: bump to 0.0.212-20260902135855-3e34d704 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1401 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:43:13] (03update) 10raymond-ndibe: components-api: bump to 0.0.212-20260902135855-3e34d704 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1401 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:43:35] (03update) 10raymond-ndibe: builds-builder: bump to 0.0.153-20260902135338-e7c94e34 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1402 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:43:37] (03approved) 10raymond-ndibe: builds-builder: bump to 0.0.153-20260902135338-e7c94e34 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1402 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:43:37] (03update) 10raymond-ndibe: builds-builder: bump to 0.0.153-20260902135338-e7c94e34 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1402 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:43:51] (03merge) 10raymond-ndibe: components-api: bump to 0.0.212-20260902135855-3e34d704 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1401 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:43:53] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.94-20260902135935-991a08d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1400 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:43:57] (03approved) 10raymond-ndibe: jobs-emailer: bump to 0.0.94-20260902135935-991a08d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1400 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:43:59] (03update) 10raymond-ndibe: jobs-emailer: bump to 0.0.94-20260902135935-991a08d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1400 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:44:11] (03update) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.202-20260902134959-47547230 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1403 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:44:16] (03approved) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.202-20260902134959-47547230 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1403 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:44:29] (03update) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.202-20260902134959-47547230 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1403 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:44:35] (03update) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.202-20260902134959-47547230 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1403 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:44:39] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.90-20260902135022-e09fd871 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1398 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:44:43] (03approved) 10raymond-ndibe: ingress-admission: bump to 0.0.90-20260902135022-e09fd871 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1398 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:44:50] (03merge) 10raymond-ndibe: jobs-emailer: bump to 0.0.94-20260902135935-991a08d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1400 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:44:54] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.90-20260902135022-e09fd871 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1398 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:45:06] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.90-20260902135022-e09fd871 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1398 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:45:14] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.90-20260902135022-e09fd871 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1398 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:45:24] (03update) 10raymond-ndibe: volume-admission: bump to 0.0.88-20260902135156-b9eeaaf1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1404 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:45:26] (03approved) 10raymond-ndibe: volume-admission: bump to 0.0.88-20260902135156-b9eeaaf1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1404 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:45:37] (03update) 10raymond-ndibe: volume-admission: bump to 0.0.88-20260902135156-b9eeaaf1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1404 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:45:42] (03update) 10raymond-ndibe: volume-admission: bump to 0.0.88-20260902135156-b9eeaaf1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1404 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:46:10] (03update) 10raymond-ndibe: builds-builder: bump to 0.0.153-20260902135338-e7c94e34 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1402 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:46:23] (03update) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.202-20260902134959-47547230 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1403 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:47:01] (03merge) 10raymond-ndibe: volume-admission: bump to 0.0.88-20260902135156-b9eeaaf1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1404 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:47:19] (03update) 10raymond-ndibe: builds-builder: bump to 0.0.153-20260902135338-e7c94e34 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1402 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:47:25] (03update) 10raymond-ndibe: builds-builder: bump to 0.0.153-20260902135338-e7c94e34 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1402 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:48:01] (03merge) 10raymond-ndibe: builds-builder: bump to 0.0.153-20260902135338-e7c94e34 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1402 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:48:19] (03update) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.202-20260902134959-47547230 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1403 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:48:28] (03update) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.202-20260902134959-47547230 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1403 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:48:45] (03update) 10raymond-ndibe: builds-api: bump to 0.0.221-20260902135258-7a258309 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1405 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:48:49] (03approved) 10raymond-ndibe: builds-api: bump to 0.0.221-20260902135258-7a258309 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1405 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:48:56] (03update) 10raymond-ndibe: builds-api: bump to 0.0.221-20260902135258-7a258309 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1405 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:49:10] (03merge) 10raymond-ndibe: maintain-kubeusers: bump to 0.0.202-20260902134959-47547230 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1403 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:49:24] (03update) 10raymond-ndibe: builds-api: bump to 0.0.221-20260902135258-7a258309 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1405 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:49:52] (03merge) 10raymond-ndibe: builds-api: bump to 0.0.221-20260902135258-7a258309 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1405 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:50:01] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.107-20260902140532-1881747b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1406 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:50:04] (03approved) 10raymond-ndibe: api-gateway: bump to 0.0.107-20260902140532-1881747b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1406 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:50:07] (03update) 10raymond-ndibe: api-gateway: bump to 0.0.107-20260902140532-1881747b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1406 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:50:45] (03merge) 10raymond-ndibe: api-gateway: bump to 0.0.107-20260902140532-1881747b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1406 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:50:47] (03update) 10raymond-ndibe: envvars-admission: bump to 0.0.42-20260902135146-45240b57 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1407 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:50:50] (03approved) 10raymond-ndibe: envvars-admission: bump to 0.0.42-20260902135146-45240b57 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1407 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:50:53] (03PS1) 10Krinkle: Config: Remove `disableClassifyEditor` setting [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334281 [03:50:54] (03update) 10raymond-ndibe: envvars-admission: bump to 0.0.42-20260902135146-45240b57 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1407 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:51:22] (03merge) 10raymond-ndibe: envvars-admission: bump to 0.0.42-20260902135146-45240b57 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1407 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:51:22] (03update) 10raymond-ndibe: registry-admission: bump to 0.0.79-20260902140719-e192d371 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1408 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:51:25] (03approved) 10raymond-ndibe: registry-admission: bump to 0.0.79-20260902140719-e192d371 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1408 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:51:30] (03update) 10raymond-ndibe: registry-admission: bump to 0.0.79-20260902140719-e192d371 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1408 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:51:59] (03CR) 10CI reject: [V:04-1] Config: Remove `disableClassifyEditor` setting [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334281 (owner: 10Krinkle) [03:52:06] (03merge) 10raymond-ndibe: registry-admission: bump to 0.0.79-20260902140719-e192d371 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1408 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:52:52] (03PS2) 10Krinkle: Config: Remove `disableClassifyEditor` setting [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334281 [03:52:58] (03CR) 10Krinkle: [C:03+2] Config: Remove `disableClassifyEditor` setting [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334281 (owner: 10Krinkle) [03:53:27] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.74-20260902135358-6d48c550 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1409 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:53:28] (03approved) 10raymond-ndibe: maintain-harbor: bump to 0.0.74-20260902135358-6d48c550 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1409 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:53:28] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.74-20260902135358-6d48c550 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1409 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:54:23] (03Merged) 10jenkins-bot: Config: Remove `disableClassifyEditor` setting [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334281 (owner: 10Krinkle) [03:54:28] (03merge) 10raymond-ndibe: maintain-harbor: bump to 0.0.74-20260902135358-6d48c550 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1409 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:54:35] (03update) 10raymond-ndibe: wmcs-k8s-metrics: bump to 0.0.25-20260902141113-35f68462 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1410 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:54:38] (03approved) 10raymond-ndibe: wmcs-k8s-metrics: bump to 0.0.25-20260902141113-35f68462 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1410 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:54:41] (03update) 10raymond-ndibe: wmcs-k8s-metrics: bump to 0.0.25-20260902141113-35f68462 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1410 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:55:04] (03PS8) 10Krinkle: Release 5.0.0 [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334182 [03:55:23] (03merge) 10raymond-ndibe: wmcs-k8s-metrics: bump to 0.0.25-20260902141113-35f68462 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1410 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:55:38] (03PS9) 10Krinkle: Release 5.0.0 [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334182 [03:55:40] (03update) 10raymond-ndibe: ingress-admission: bump to 0.0.90-20260902135022-e09fd871 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1398 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [03:56:18] (03CR) 10Krinkle: [C:03+2] Release 5.0.0 [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334182 (owner: 10Krinkle) [03:57:34] (03update) 10raymond-ndibe: global: add public tool search with config endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/186 (https://phabricator.wikimedia.org/T433690) (owner: 10dcaro) [03:57:47] (03Merged) 10jenkins-bot: Release 5.0.0 [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334182 (owner: 10Krinkle) [03:58:47] (03update) 10raymond-ndibe: envvars-admission: Drop old ElasticSearch-related variables [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1394 (https://phabricator.wikimedia.org/T401818) (owner: 10taavi) [03:59:15] (03update) 10raymond-ndibe: envvars-admission: Drop old ElasticSearch-related variables [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1394 (https://phabricator.wikimedia.org/T401818) (owner: 10taavi) [03:59:17] (03approved) 10raymond-ndibe: envvars-admission: Drop old ElasticSearch-related variables [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1394 (https://phabricator.wikimedia.org/T401818) (owner: 10taavi) [03:59:55] (03update) 10raymond-ndibe: cancel: split latest deployment endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/182 (https://phabricator.wikimedia.org/T434499) (owner: 10mahveotm) [04:01:11] (03approved) 10raymond-ndibe: tools: Remove Elasticsearch DNS record [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/125 (https://phabricator.wikimedia.org/T401818) (owner: 10taavi) [04:01:15] (03update) 10raymond-ndibe: tools: Remove Elasticsearch DNS record [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/125 (https://phabricator.wikimedia.org/T401818) (owner: 10taavi) [04:02:25] (03PS1) 10Krinkle: docs: Add release note for URL encoding fix [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334287 [04:02:49] (03CR) 10Krinkle: [C:03+2] docs: Add release note for URL encoding fix [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334287 (owner: 10Krinkle) [04:02:57] (03CR) 10Krinkle: [C:03+2] ListManager: Tests for combining wl, al, and bot for the same user [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334044 (https://phabricator.wikimedia.org/T327129) (owner: 10Krinkle) [04:03:03] (03CR) 10Krinkle: [C:03+2] ListManager: Fix adding to admin/bot list while on WL [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334149 (https://phabricator.wikimedia.org/T327129) (owner: 10Krinkle) [04:04:04] (03Merged) 10jenkins-bot: docs: Add release note for URL encoding fix [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334287 (owner: 10Krinkle) [04:04:16] (03Merged) 10jenkins-bot: ListManager: Tests for combining wl, al, and bot for the same user [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334044 (https://phabricator.wikimedia.org/T327129) (owner: 10Krinkle) [04:04:18] (03Merged) 10jenkins-bot: ListManager: Fix adding to admin/bot list while on WL [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334149 (https://phabricator.wikimedia.org/T327129) (owner: 10Krinkle) [04:06:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:11:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:15:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:16:28] 10Cloud-VPS, 06tools-infrastructure-team, 10Ceph: "osd.224 observed stalled read indications in DB device" - https://phabricator.wikimedia.org/T434402#12284043 (10Andrew) osd.54 observed stalled read indications in DB device osd.334 observed stalled read indications in DB device osd.335 observ... [04:20:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:38:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:42:31] (03merge) 10raymond-ndibe: ingress-admission: bump to 0.0.90-20260902135022-e09fd871 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1398 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [04:42:42] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.568-20260902141641-99c9c4e5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1397 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [04:42:44] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.568-20260902141641-99c9c4e5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1397 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [04:42:45] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.568-20260902141641-99c9c4e5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1397 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [04:43:14] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.568-20260902141641-99c9c4e5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1397 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [04:43:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:44:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:49:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:50:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:55:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:00:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:02:43] (03update) 10raymond-ndibe: refactor: Generalize job event models and push format [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [05:03:00] (03update) 10raymond-ndibe: refactor: Generalize job event models and push format [repos/cloud/toolforge/logs-api] (write-refactor-for-read) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [05:03:16] (03update) 10raymond-ndibe: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [05:05:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:10:41] (03update) 10raymond-ndibe: refactor: Generalize job event models and push format [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [05:13:39] (03CR) 10Hashar: "I have manually triggered the coverage job using:" [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334149 (https://phabricator.wikimedia.org/T327129) (owner: 10Krinkle) [05:29:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:34:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:44:56] (03PS1) 10Krinkle: docs: Add link to Wikidata lag in release notes [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334556 (https://phabricator.wikimedia.org/T327135) [06:52:00] (03open) 10countcount: Remove the senior-code-reviewer agent and a dangling doc pointer [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/43 [06:59:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:04:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:10:53] (03PS1) 10Krinkle: Project: Refactor double-negative nonStrict into bool strict [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334593 [07:11:05] (03CR) 10Krinkle: [C:03+2] docs: Add link to Wikidata lag in release notes [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334556 (https://phabricator.wikimedia.org/T327135) (owner: 10Krinkle) [07:11:09] (03CR) 10Krinkle: [C:03+2] Project: Refactor double-negative nonStrict into bool strict [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334593 (owner: 10Krinkle) [07:12:20] (03Merged) 10jenkins-bot: docs: Add link to Wikidata lag in release notes [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334556 (https://phabricator.wikimedia.org/T327135) (owner: 10Krinkle) [07:14:14] (03Merged) 10jenkins-bot: Project: Refactor double-negative nonStrict into bool strict [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334593 (owner: 10Krinkle) [07:42:58] (03PS1) 10Krinkle: RCReader: Fix malformed title in protect and modifyprotect events [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334628 (https://phabricator.wikimedia.org/T327133) [07:55:29] (03PS2) 10Krinkle: RCReader: Fix malformed title in protect and modifyprotect events [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334628 (https://phabricator.wikimedia.org/T327133) [07:59:55] (03CR) 10Krinkle: [C:03+2] RCReader: Fix malformed title in protect and modifyprotect events [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334628 (https://phabricator.wikimedia.org/T327133) (owner: 10Krinkle) [08:02:56] (03Merged) 10jenkins-bot: RCReader: Fix malformed title in protect and modifyprotect events [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334628 (https://phabricator.wikimedia.org/T327133) (owner: 10Krinkle) [08:15:06] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [08:15:16] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [08:15:18] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [08:15:33] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [08:15:41] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [08:15:58] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24 [08:16:06] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [08:17:15] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [08:17:23] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [08:18:38] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41 [08:19:21] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [08:31:50] (03PS1) 10Krinkle: Messages: Fix loss of previous messages if msgs reload fails [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334720 [08:31:50] (03PS1) 10Krinkle: Project: Refactor internals to refer to MW message keys by their lcfirst form [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334721 [08:33:12] (03CR) 10CI reject: [V:04-1] Messages: Fix loss of previous messages if msgs reload fails [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334720 (owner: 10Krinkle) [08:33:18] (03CR) 10CI reject: [V:04-1] Project: Refactor internals to refer to MW message keys by their lcfirst form [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334721 (owner: 10Krinkle) [08:35:33] (03merge) 10taavi: tools: Remove Elasticsearch DNS record [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/125 (https://phabricator.wikimedia.org/T401818) [08:35:35] (03update) 10taavi: tools: Remove Elasticsearch volumes [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/126 (https://phabricator.wikimedia.org/T401818) [08:37:26] (03PS1) 10Krinkle: Program: Fix "reload" command to save Projects.xml, like "batchreload" [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334724 [08:38:38] (03CR) 10CI reject: [V:04-1] Program: Fix "reload" command to save Projects.xml, like "batchreload" [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334724 (owner: 10Krinkle) [08:47:04] (03PS2) 10Krinkle: Messages: Fix loss of previous messages if msgs reload fails [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334720 [08:47:04] (03PS2) 10Krinkle: Project: Refactor internals to refer to MW message keys by their lcfirst form [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334721 [08:47:06] (03PS2) 10Krinkle: Program: Fix "reload" command to save Projects.xml, like "batchreload" [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334724 [08:59:02] (03CR) 10Krinkle: [C:03+2] Messages: Fix loss of previous messages if msgs reload fails [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334720 (owner: 10Krinkle) [08:59:07] (03CR) 10Krinkle: [C:03+2] Project: Refactor internals to refer to MW message keys by their lcfirst form [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334721 (owner: 10Krinkle) [08:59:10] (03CR) 10Krinkle: [C:03+2] Program: Fix "reload" command to save Projects.xml, like "batchreload" [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334724 (owner: 10Krinkle) [09:01:01] (03Merged) 10jenkins-bot: Messages: Fix loss of previous messages if msgs reload fails [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334720 (owner: 10Krinkle) [09:01:02] (03Merged) 10jenkins-bot: Project: Refactor internals to refer to MW message keys by their lcfirst form [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334721 (owner: 10Krinkle) [09:01:03] (03Merged) 10jenkins-bot: Program: Fix "reload" command to save Projects.xml, like "batchreload" [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334724 (owner: 10Krinkle) [09:12:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:14:17] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [09:14:17] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [09:14:24] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [09:14:26] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [09:14:33] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24 [09:14:34] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [09:15:30] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [09:15:31] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [09:16:12] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41 [09:16:13] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [09:17:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:25:02] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33738502759 (https://github.com/cluebotng/component-configs/commits/a57be2aa8221abd40bf701a3d617fddcec5fd47b) [09:25:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [09:30:34] (03PS1) 10Krinkle: Program: Remove send_message_multi in favor of IrcClient.send_message [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334750 [09:32:00] (03PS2) 10Krinkle: Program: Remove send_message_multi in favor of IrcClient.send_message [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1334750 [09:54:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:59:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:04:43] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33742087870 (https://github.com/cluebotng/component-configs/commits/781284a9a6b62d000c7c570fb34d20bbe21933a0) [10:04:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [10:14:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:24:38] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:28:53] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12284898 (10BTullis) I checked on https://stream-internal.wikimedia.org/v2/u... [10:37:56] (03PS1) 10Gergő Tisza: frontend: Update results object when getting more results [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1334798 [10:39:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:54:40] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12285330 (10BTullis) We now have data in the hive table. ` spark-sql (defaul... [12:01:59] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12285377 (10elukey) @Jclark-ctr the new change is merged, I was able to provision 1055 and 1056, but 1053/54 seem not reachable yet so I am not sure how to... [12:13:59] (03update) 10samwilson: Upgrade to Symfony 8.1 [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/12 (https://phabricator.wikimedia.org/T432808) [12:22:41] (03update) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [12:22:46] (03update) 10tlepage: refactor: Generalize job event models and push format [repos/cloud/toolforge/logs-api] (write-refactor-for-read) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [12:22:50] (03update) 10tlepage: refactor: Generalize job event models and push format [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) [12:26:04] (03open) 10tlepage: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] (write-refactor-for-read) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) [12:26:41] (03update) 10tlepage: refactor: Generalize job event models [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) [12:27:03] (03update) 10tlepage: refactor: Generalize job event models and push format [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [12:27:22] (03update) 10tlepage: refactor: Generalize job event models and push format [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [12:27:48] (03update) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [12:28:05] (03update) 10tlepage: refactor: Generalize job event models and push format [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [12:29:02] (03update) 10samwilson: Upgrade to Symfony 8.1 [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/12 (https://phabricator.wikimedia.org/T432808) [12:31:10] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12285520 (10fgiunchedi) >>! In T435921#12282448, @VRiley-WMF wrote: > @fgiunchedi while trying to image the first server... [12:46:18] (03update) 10tlepage: refactor: Generalize job event models [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) [12:57:05] (03merge) 10fnegri: alerts-api: Add initial scaffolding [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/1 (https://phabricator.wikimedia.org/T435971) [13:01:09] 10Toolforge, 06tools-platform-team: [all] first of the month dependency upgrades for september - https://phabricator.wikimedia.org/T436796#12285720 (10Raymond_Ndibe) 05In progress→03Resolved [13:03:16] (03update) 10raymond-ndibe: models: support webservice job-type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [13:09:03] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12285768 (10Andrew) I'm going to see if I can do controller firmware updates on 1045-1047 [13:13:43] PROBLEM - Host cloudcephosd1045 is DOWN: PING CRITICAL - Packet loss = 100% [13:13:51] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12285795 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1... [13:14:04] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T348755 https://phabricator.wikimedia.org/T428898) [13:15:27] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T428898) [13:15:43] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12285801 (10Andrew) ` current version 1+dz02 is ahead of target version 1+dl7c, use force to downgrade ` so... I'm going... [13:16:07] RECOVERY - Host cloudcephosd1045 is UP: PING OK - Packet loss = 0%, RTA = 0.36 ms [13:22:46] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12285823 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1003 for host cloudcephosd1055.eqiad.wmnet with OS bookworm [13:28:50] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12285875 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudv... [13:29:42] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12285878 (10VRiley-WMF) [13:36:42] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 3 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12285914 (10fgiunchedi) [13:37:39] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 3 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12285925 (10fgiunchedi) >>! In T435921#12282448, @VRiley-WMF wrote: > @fgiunchedi while trying to image the first server... [13:40:44] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 3 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12285939 (10VRiley-WMF) Understood. It's strange because when I log into the console, it seems like it finishes (went to... [13:51:33] (03update) 10fnegri: cli: use new status field instead of status_short/long [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/178 (https://phabricator.wikimedia.org/T430904) (owner: 10raymond-ndibe) [13:51:41] (03approved) 10fnegri: cli: use new status field instead of status_short/long [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/178 (https://phabricator.wikimedia.org/T430904) (owner: 10raymond-ndibe) [13:51:43] (03update) 10fnegri: cli: use new status field instead of status_short/long [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/178 (https://phabricator.wikimedia.org/T430904) (owner: 10raymond-ndibe) [13:52:31] (03update) 10fnegri: cli: use new status field instead of status_short/long [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/178 (https://phabricator.wikimedia.org/T430904) (owner: 10raymond-ndibe) [14:01:52] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12286061 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudv... [14:02:40] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 3 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12286074 (10fgiunchedi) >>! In T435921#12285939, @VRiley-WMF wrote: > Understood. It's strange because when I log into t... [14:04:00] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 3 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12286079 (10VRiley-WMF) Gotcha, no problem. It looks like 1067 went through (it's still finishing up) but I will rerun i... [14:06:05] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12286086 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1... [14:06:26] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 3 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12286089 (10VRiley-WMF) [14:08:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:08:59] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12286102 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1003 for host cloudcephosd1055.eqiad.wmne... [14:09:10] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12286103 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1003 for host cloudcephosd1055.eqiad.... [14:15:10] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate utrs away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435787#12286120 (10AmandaNP) The production server is migrated and is in shutdown. Alpha was just deleted - i'll remake it at a later date. I will try and handle the database... [14:16:45] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12286125 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1003 for host cloudcephosd1055.eqiad.wmne... [14:30:00] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 3 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12286173 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1066.... [14:30:28] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 3 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12286177 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1... [14:38:51] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12286240 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1... [14:39:29] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: New Cloud VPS/Toolforge dev account (VGois) cannot SSH to bastion - key correctly registered but rejected - https://phabricator.wikimedia.org/T436946 (10VGois) 03NEW [14:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:48:49] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12286301 (10VRiley-WMF) [14:49:49] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: New Cloud VPS/Toolforge dev account (VGois) cannot SSH to bastion - key correctly registered but rejected - https://phabricator.wikimedia.org/T436946#12286308 (10taavi) 05Open→03Invalid According to the logs, you are tryi... [14:50:10] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of E4 and into C8 - https://phabricator.wikimedia.org/T431682#12286310 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudv... [14:52:58] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: New Cloud VPS/Toolforge dev account (VGois) cannot SSH to bastion - key correctly registered but rejected - https://phabricator.wikimedia.org/T436946#12286319 (10VGois) >>! In T436946#12286308, @taavi wrote: > According to th... [14:56:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:57:22] 10Toolforge (Quota-requests), 06tools-infrastructure-team: Request increased quota for wiki-economics Toolforge tool - https://phabricator.wikimedia.org/T436614#12286339 (10fnegri) @Schiste we just discussed this in a meeting with #tools-platform-team and #tools-infrastructure-team , the consensus is that ~200... [15:05:44] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12286394 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1066.... [15:06:41] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12286401 (10VRiley-WMF) [15:11:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:17:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:22:38] RESOLVED: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:47:12] 10Tool-wikinewsie: Wikinewsie suggested headline from AI recap - https://phabricator.wikimedia.org/T436419#12286597 (10Chimaifeanyi29) a:03Chimaifeanyi29 [15:48:36] 10Tool-wikimonitor, 07patch-welcome: Saved edit summaries for faster undo and rollback actions - https://phabricator.wikimedia.org/T428349#12286600 (10Gerges) [15:52:46] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12286608 (10BTullis) [15:54:57] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12286617 (10BTullis) 05In progress→03Resolved I think that this is a... [16:02:54] 10Cloud-VPS (Project-requests), 06Data-Platform-SRE: Request creation of cephvmm VPS project - https://phabricator.wikimedia.org/T436647#12286661 (10BLiviero-WMF) 05Open→03Declined Thank you all for weighing in on this project and ask. Following @Gehel 's guidance, we will table this ask. Please fee... [16:56:08] (03update) 10fnegri: status: job with failing healthcheck should not be marked running [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/376 (https://phabricator.wikimedia.org/T436726) (owner: 10raymond-ndibe) [17:12:12] (03update) 10fnegri: status: job with failing healthcheck should not be marked running [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/376 (https://phabricator.wikimedia.org/T436726) (owner: 10raymond-ndibe) [17:12:12] (03approved) 10fnegri: status: job with failing healthcheck should not be marked running [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/376 (https://phabricator.wikimedia.org/T436726) (owner: 10raymond-ndibe) [18:00:48] 10Cloud-VPS, 06tools-infrastructure-team: Need raid fw files on cumin to upgrade cloudcephosd1045-1047 - https://phabricator.wikimedia.org/T436966 (10Andrew) 03NEW [18:01:53] 10Cloud-VPS, 06tools-infrastructure-team: Need raid fw files on cumin to upgrade cloudcephosd1045-1047 - https://phabricator.wikimedia.org/T436966#12287086 (10Andrew) Today my goal is H755 controller 52.26.0-5179 -> 52.38.0-6897 but I'll take the latest, whatever it is. [18:17:22] 10Cloud-VPS, 06tools-infrastructure-team: Need raid fw files on cumin to upgrade cloudcephosd1045-1047 - https://phabricator.wikimedia.org/T436966#12287190 (10RobH) Pushed SAS-RAID_Firmware_2WJX5_WN64_52.38.0-6897_A13_01.EXE into the R660 storage directory on cumin1003. should be able to run and see it there a... [19:21:00] 10Tool-wikinewsie: Adjustable image alignment on Wikinewsie - https://phabricator.wikimedia.org/T434467#12287435 (10Pharos) The new default fallback works really well for photos, but sometimes produces strange results for illustrations like logos and flags. So a manual crop tool may still be useful, or even just... [19:31:53] (03update) 10waldir: T432938: Add interface language switcher with i18n loading mechanism [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/9 (https://phabricator.wikimedia.org/T432938) (owner: 10chimaifeanyi29) [19:34:27] (03approved) 10waldir: T432938: Add interface language switcher with i18n loading mechanism [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/9 (https://phabricator.wikimedia.org/T432938) (owner: 10chimaifeanyi29) [19:35:04] (03update) 10waldir: T432938: Add interface language switcher with i18n loading mechanism [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/9 (https://phabricator.wikimedia.org/T432938) (owner: 10chimaifeanyi29) [19:37:09] 10Tool-wikinewsie: User ability to promote non-categorized Wikinewsie stories to Front Page portal - https://phabricator.wikimedia.org/T436982 (10Pharos) 03NEW [19:38:32] (03merge) 10waldir: T432938: Add interface language switcher with i18n loading mechanism [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/9 (https://phabricator.wikimedia.org/T432938) (owner: 10chimaifeanyi29) [20:01:38] 10Tool-Pageviews, 06Data-Engineering, 06Data-Engineering-Icebox, 07User-notice: Pageviews Analysis 2026 (Vue + Codex) - https://phabricator.wikimedia.org/T378549#12287581 (10MusikAnimal) I think we should advertise in Tech News, especially given the new features. I've added an entry: https://meta.wikim... [20:07:33] PROBLEM - SSH on cloudvirt1045 is CRITICAL: connect to address 10.64.20.11 and port 22: Connection refused https://wikitech.wikimedia.org/wiki/SSH/monitoring [20:08:38] FIRING: [3x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:08:50] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1045 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [20:09:14] FIRING: [6x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-gateway-1.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [20:11:59] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [20:12:03] FIRING: InstanceDown: Project toolsbeta instance toolsbeta-test-k8s-worker-nfs-9 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:12:03] FIRING: [3x] InstanceDown: Project tools instance tools-k8s-worker-108 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:12:03] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [20:13:47] FIRING: NodeDown: Node cloudcephosd1045 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1045 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:14:33] RECOVERY - SSH on cloudvirt1045 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u4 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [20:18:47] RESOLVED: NodeDown: Node cloudcephosd1045 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1045 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:28:19] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1045 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [20:28:29] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [20:28:33] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [20:28:38] RESOLVED: [8x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:30004 has failed probes (http_infra_tracing_loki_svc_toolsbeta_eqiad1_wikimedia_cloud_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:29:33] RESOLVED: [3x] InstanceDown: Project tools instance tools-k8s-worker-108 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:29:33] RESOLVED: InstanceDown: Project toolsbeta instance toolsbeta-test-k8s-worker-nfs-9 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:36:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:51:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:14:55] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [21:15:05] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [21:15:09] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [21:15:21] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [21:15:29] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [21:15:30] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [21:15:46] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24 [21:15:55] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [21:15:56] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [21:17:00] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [21:17:09] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [21:17:12] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [21:18:21] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41 [21:19:01] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [21:19:03] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [22:14:16] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [22:14:16] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [22:14:24] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [22:14:25] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [22:14:33] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24 [22:14:34] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [22:15:31] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [22:15:31] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [22:16:14] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41 [22:16:14] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [23:06:12] (03close) 10habs: Fix T436414: Prevent external link rendering in Wikinewsie AI text [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/11 (owner: 10akaza24) [23:06:12] (03update) 10habs: Fix T436414: Prevent external link rendering in Wikinewsie AI text [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/11 (owner: 10akaza24) [23:14:29] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [23:14:38] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [23:14:54] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [23:15:01] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [23:15:19] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24 [23:15:26] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [23:16:35] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [23:16:43] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [23:18:01] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41 [23:18:44] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [23:24:51] (03update) 10habs: T435732: Add tooltip explanations for Mix, Editors, Edits [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/10 (https://phabricator.wikimedia.org/T435732) (owner: 10chimaifeanyi29) [23:25:28] (03merge) 10habs: T435732: Add tooltip explanations for Mix, Editors, Edits [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/10 (https://phabricator.wikimedia.org/T435732) (owner: 10chimaifeanyi29)