[00:14:14] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/17 [00:14:24] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/29 [00:14:39] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/24 [00:15:38] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/56 [00:17:02] (03update) 10renovatebot: Update pnpm to v11.25.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/41 [01:36:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:41:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:45:53] 10Cloud-VPS (Project-requests), 06MediaWiki-Media-Platform-Team: Request creation of imagor VPS project - https://phabricator.wikimedia.org/T437088#12293498 (10Samwilson) Should we name the project after the team, e.g. `mediaplatform`, so we can use it for other experiments in the future? [01:57:43] FIRING: NodeTextfileStale: Stale textfile for clouddumps1002:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [02:00:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:04:51] (03update) 10raymond-ndibe: refactor: Generalize job event models and push format [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [02:05:32] (03update) 10raymond-ndibe: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] (write-refactor-for-read) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [02:06:02] (03update) 10raymond-ndibe: refactor: Generalize job event models [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) (owner: 10tlepage) [02:15:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:22:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:27:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:34:23] (03open) 10dreamyshade: Shift focus to list-making [toolforge-repos/findlinkfast] - 10https://gitlab.wikimedia.org/toolforge-repos/findlinkfast/-/merge_requests/2 [02:38:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:43:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:44:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:49:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:04:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:05:22] (03update) 10dreamyshade: Shift focus to list-making [toolforge-repos/findlinkfast] - 10https://gitlab.wikimedia.org/toolforge-repos/findlinkfast/-/merge_requests/2 [03:08:10] (03update) 10dreamyshade: Shift focus to list-making [toolforge-repos/findlinkfast] - 10https://gitlab.wikimedia.org/toolforge-repos/findlinkfast/-/merge_requests/2 [03:09:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:43:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:48:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:57:44] FIRING: NodeTextfileStale: Stale textfile for clouddumps1002:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [06:08:26] 10Cloud-VPS (Project-requests), 06MediaWiki-Media-Platform-Team: Request creation of imagor VPS project - https://phabricator.wikimedia.org/T437088#12293534 (10Aklapper) [06:29:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:34:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:40:53] (03open) 10arnaudb: gather_precommit: send a User-Agent on the group listing request [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/98 (https://phabricator.wikimedia.org/T425441) [07:20:14] (03approved) 10dcaro: gather_precommit: send a User-Agent on the group listing request [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/98 (https://phabricator.wikimedia.org/T425441) (owner: 10arnaudb) [07:20:20] (03merge) 10dcaro: gather_precommit: send a User-Agent on the group listing request [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/98 (https://phabricator.wikimedia.org/T425441) (owner: 10arnaudb) [07:27:29] RESOLVED: NodeTextfileStale: Stale textfile for clouddumps1002:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [08:11:08] (03update) 10dcaro: Silly commit [toolforge-repos/wm-lol] - 10https://gitlab.wikimedia.org/toolforge-repos/wm-lol/-/merge_requests/1 [08:11:11] (03update) 10dcaro: Silly commit [toolforge-repos/wm-lol] - 10https://gitlab.wikimedia.org/toolforge-repos/wm-lol/-/merge_requests/1 [09:18:47] (03approved) 10dcaro: Fix task template division by zero and API failure display [toolforge-repos/cloudvps-quota] - 10https://gitlab.wikimedia.org/toolforge-repos/cloudvps-quota/-/merge_requests/3 (https://phabricator.wikimedia.org/T433836 https://phabricator.wikimedia.org/T433837) (owner: 10filippo) [09:42:31] 10Cloud-VPS (Project-requests), 06MediaWiki-Media-Platform-Team: Request creation of imagor VPS project - https://phabricator.wikimedia.org/T437088#12294152 (10Ladsgroup) >>! In T437088#12293498, @Samwilson wrote: > Should we name the project after the team, e.g. `mediaplatform`, so we can use it for other exp... [09:44:21] 10Cloud-VPS (Project-requests), 06MediaWiki-Media-Platform-Team: Request creation of imagor VPS project - https://phabricator.wikimedia.org/T437088#12294157 (10Samwilson) Yes, whatever they prefer is fine by me! [09:45:34] 10Cloud-VPS (Project-requests), 06MediaWiki-Media-Platform-Team: Request creation of imagor VPS project - https://phabricator.wikimedia.org/T437088#12294164 (10taavi) Please see the "Project scope" heading on #cloud-vps-project-requests and the Umbrella projects entry on https://wikitech.wikimedia.org/wiki/Hel... [10:03:25] (03update) 10tlepage: refactor: Generalize job event models and push format [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [10:03:36] (03update) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [10:03:39] (03update) 10tlepage: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] (write-refactor-for-read) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) [10:03:39] (03update) 10tlepage: refactor: Generalize job event models [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) [10:06:50] (03update) 10tlepage: refactor: Generalize job event models [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) [10:07:04] (03update) 10tlepage: refactor: Generalize job event models [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) [10:09:48] (03update) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [10:16:59] 10Data-Services, 06tools-platform-team: Update Wiki Replicas documentation for x4 rollout - https://phabricator.wikimedia.org/T437208 (10taavi) 03NEW [10:17:23] 10Data-Services, 06tools-platform-team: Update Wiki Replicas documentation for x4 rollout - https://phabricator.wikimedia.org/T437208#12294256 (10taavi) [10:17:29] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 13Patch-For-Review: Add support for x1 and x4 sections on wiki replicas on the load balancer layer - https://phabricator.wikimedia.org/T409560#12294258 (10taavi) [10:17:32] 10Data-Services, 06tools-platform-team: Update Wiki Replicas documentation for x4 rollout - https://phabricator.wikimedia.org/T437208#12294261 (10taavi) [10:34:09] 10Cloud-VPS (Project-requests), 06MediaWiki-Media-Platform-Team: Request creation of imagor VPS project - https://phabricator.wikimedia.org/T437088#12294333 (10Ladsgroup) If you prefer non-umberlla (ella ella ella eh) projects, we can go with imagor then. [10:37:10] 10Data-Services, 06tools-platform-team, 13Patch-For-Review: Update Wiki Replicas documentation for x4 rollout - https://phabricator.wikimedia.org/T437208#12294337 (10taavi) ` 2026-09-07T10:35:40Z __main__ INFO : Syncing additional CNAME: 'links.commonswiki.analytics.db.svc.wikimedia.cloud.' 2026-09-07... [10:54:28] (03update) 10taavi: sql: Add support for links cluster [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/23 (https://phabricator.wikimedia.org/T437208) [10:54:28] (03update) 10taavi: sql: Document flags on man page [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/24 [10:54:40] (03open) 10taavi: sql: Document flags on man page [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/24 [10:54:40] (03open) 10taavi: sql: Add support for links cluster [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/23 (https://phabricator.wikimedia.org/T437208) [10:55:30] (03update) 10taavi: sql: Add support for links cluster [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/23 (https://phabricator.wikimedia.org/T437208) [10:55:34] (03update) 10taavi: sql: Add support for links cluster [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/23 (https://phabricator.wikimedia.org/T437208) [10:55:44] (03update) 10taavi: sql: Document flags on man page [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/24 [10:55:47] (03update) 10taavi: sql: Document flags on man page [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/24 [11:05:02] 10Toolforge, 06tools-platform-team: Missing space in "no deployments found" message - https://phabricator.wikimedia.org/T437212 (10taavi) 03NEW [11:12:58] (03update) 10taavi: sql: Document flags on man page [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/24 [11:35:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:40:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:00:32] (03update) 10renovatebot: Update vitest monorepo to v5 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/30 [12:02:20] (03merge) 10countcount: Update vitest monorepo to v5 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/30 (owner: 10renovatebot) [12:02:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:05:52] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [12:07:32] 10Toolforge, 06tools-platform-team: tool registration failed AKA delete tool wikiblame and/or replace it with a copy of tool blame - https://phabricator.wikimedia.org/T437214 (10Flominator) 03NEW [12:07:36] (03merge) 10countcount: Update vitest monorepo to v5 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/42 (owner: 10renovatebot) [12:12:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:14:11] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [12:14:24] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [12:14:37] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [12:15:41] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [12:17:13] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [12:19:47] 10Toolforge, 06tools-infrastructure-team: Stand up an etcd cluster for requestctl usage - https://phabricator.wikimedia.org/T410721#12294565 (10taavi) [12:23:59] 10Cloud-VPS, 06tools-infrastructure-team: Stand up an etcd cluster for requestctl usage - https://phabricator.wikimedia.org/T410721#12294569 (10taavi) [12:24:45] 06cloud-services-team, 06Collaboration-Services, 10GitLab, 06Machine-Learning-Team, and 4 others: Set an identifiable User-Agent on WMF tools calling gitlab.wikimedia.org - https://phabricator.wikimedia.org/T437216 (10ABran-WMF) 03NEW [12:31:56] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/44 [12:34:42] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: Enable use of requestctl for filtering Toolforge and Cloud VPS web traffic - https://phabricator.wikimedia.org/T409759#12294602 (10taavi) a:03taavi [12:34:47] 10Cloud-VPS, 06tools-infrastructure-team: Stand up an etcd cluster for requestctl usage - https://phabricator.wikimedia.org/T410721#12294606 (10taavi) a:03taavi [12:41:10] 06tools-infrastructure-team: Upgrade cloudcumin hosts to Trixie - https://phabricator.wikimedia.org/T436667#12294617 (10Volans) I've had a quick chat with Moritz and for this use case it's ok to perform an in-place upgrade if preferred. [12:54:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [12:57:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:58:38] (03update) 10dcaro: global: move `image` inside tests [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/96 (https://phabricator.wikimedia.org/T435182) [12:59:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [13:01:46] (03merge) 10filippo: global: move `image` inside tests [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/96 (https://phabricator.wikimedia.org/T435182) (owner: 10dcaro) [13:07:36] (03update) 10dcaro: global: move all stages to shared yaml [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/94 [13:08:45] (03close) 10dcaro: global: move all stages to shared yaml [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/94 [13:11:25] FIRING: [10x] SystemdUnitFailed: libvirtd-admin.socket on cloudvirt1067:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:13:40] (03approved) 10fnegri: sql: Add support for links cluster [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/23 (https://phabricator.wikimedia.org/T437208) (owner: 10taavi) [13:13:42] (03update) 10fnegri: sql: Add support for links cluster [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/23 (https://phabricator.wikimedia.org/T437208) (owner: 10taavi) [13:14:49] (03approved) 10fnegri: sql: Document flags on man page [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/24 (owner: 10taavi) [13:14:51] (03update) 10fnegri: sql: Document flags on man page [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/24 (owner: 10taavi) [13:17:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:47:07] 10Cloud-VPS, 06tools-infrastructure-team: Update ENC Hiera value column type to MEDIUMTEXT - https://phabricator.wikimedia.org/T436561#12294765 (10taavi) 05Open→03Resolved [13:47:50] 10Cloud-VPS, 06tools-infrastructure-team: Stand up an etcd cluster for requestctl usage - https://phabricator.wikimedia.org/T410721#12294768 (10taavi) [13:48:28] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: Make Toolforge etcd cookbooks more generic so they are usable for the new cloudinfra cluster - https://phabricator.wikimedia.org/T437219 (10taavi) 03NEW [13:49:27] (03update) 10tlepage: refactor: Parse Loki result streams into events [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [13:53:10] (03PS1) 10Majavah: Treat etcd clusters as separate concepts from K8s clusters [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1337603 (https://phabricator.wikimedia.org/T437219) [13:53:12] (03PS1) 10Majavah: Migrate etcd cookbooks to their own hierarchy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1337604 (https://phabricator.wikimedia.org/T437219) [13:53:15] (03PS1) 10Majavah: etcd: Use existing Enc wrapper instead of direct wmcs-enc-cli calls [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1337605 (https://phabricator.wikimedia.org/T437219) [13:54:15] (03update) 10taavi: sql: Add support for links cluster [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/23 (https://phabricator.wikimedia.org/T437208) [13:54:18] (03update) 10taavi: sql: Document flags on man page [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/24 [13:54:19] (03merge) 10taavi: sql: Add support for links cluster [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/23 (https://phabricator.wikimedia.org/T437208) [13:54:20] (03merge) 10taavi: sql: Document flags on man page [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/24 [14:01:10] (03open) 10taavi: eqiad1: Allow cloudinfra to use localdisk flavors [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/349 (https://phabricator.wikimedia.org/T410721) [14:01:59] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/349 [14:02:22] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/349 [14:02:40] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli [14:02:42] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli [14:02:49] 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, 10ops-eqiad: cloudvirt1067 missing hwraid BBU - https://phabricator.wikimedia.org/T437221 (10fgiunchedi) 03NEW [14:03:10] 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, 10ops-eqiad: cloudvirt1067 missing hwraid BBU - https://phabricator.wikimedia.org/T437221#12294886 (10fgiunchedi) Host is not in service and can be taken down at any time (with silences applied) [14:03:21] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli [14:05:03] (03CR) 10Volans: [C:03+1] "I'm not very familiar with this part of the code base but the generalization looks good, to be tested." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1337603 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [14:08:37] (03CR) 10Volans: "One question for the file move inline. In general LGTM." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1337604 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [14:09:36] (03CR) 10Volans: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1337605 (https://phabricator.wikimedia.org/T437219) (owner: 10Majavah) [14:10:47] (03approved) 10volans: eqiad1: Allow cloudinfra to use localdisk flavors [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/349 (https://phabricator.wikimedia.org/T410721) (owner: 10taavi) [14:13:26] (03update) 10dcaro: models: support webservice job-type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) (owner: 10raymond-ndibe) [14:16:51] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli [14:19:31] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli [14:19:50] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli [14:22:48] (03merge) 10taavi: eqiad1: Allow cloudinfra to use localdisk flavors [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/349 (https://phabricator.wikimedia.org/T410721) [14:22:51] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [14:23:16] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [14:24:34] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 10Ceph, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Proposed improvement: Manage CephX users via exported/collected Puppet resources - https://phabricator.wikimedia.org/T399594#12294997 (10BTullis) 05Open→03Declined I think that... [14:38:46] 10Toolforge, 06tools-platform-team: [ci] move 'image' out of the global scope - https://phabricator.wikimedia.org/T435182#12295031 (10dcaro) 05Open→03Resolved [14:41:32] 10Toolforge, 06tools-platform-team: [jobs-api,jobs-cli] When a job is restarted, show both the restart count and if it was restarted not long ago - https://phabricator.wikimedia.org/T437226 (10dcaro) 03NEW [14:43:21] (03update) 10dcaro: status: job with failing healthcheck should not be marked running [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/376 (https://phabricator.wikimedia.org/T436726) (owner: 10raymond-ndibe) [14:59:02] 06tools-infrastructure-team, 10Ceph, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): ceph: Fix the CephX key guard so that Puppet can correctly update the key material - https://phabricator.wikimedia.org/T437233 (10BTullis) 03NEW [15:00:32] (03update) 10dcaro: global: add public tool search with config endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/186 (https://phabricator.wikimedia.org/T433690) [15:24:56] (03approved) 10dcaro: global: add public tool search with config endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/186 (https://phabricator.wikimedia.org/T433690) [15:26:04] (03merge) 10dcaro: global: add public tool search with config endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/186 (https://phabricator.wikimedia.org/T433690) [15:28:08] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 10Ceph, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Proposed improvement: Manage CephX users via exported/collected Puppet resources - https://phabricator.wikimedia.org/T399594#12295280 (10brouberol) a:03BTullis [15:29:34] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.213-20260907152616-34306785 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1413 (https://phabricator.wikimedia.org/T433690) [15:32:14] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: Enable use of requestctl for filtering Toolforge and Cloud VPS web traffic - https://phabricator.wikimedia.org/T409759#12295317 (10taavi) @Joe @Raine Hi! AIUI you're the people to talk to about requestctl stuff. We are finally looking at setting up a req... [15:33:34] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [15:35:29] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [15:36:55] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [15:37:14] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [15:44:27] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [15:45:06] 06cloud-services-team, 10Quarry, 06tools-platform-team: replace ingress-nginx in Quarry - https://phabricator.wikimedia.org/T427849#12295329 (10taavi) 05Open→03Invalid Unless I am missing something, Quarry does not use ingress-nginx (or any other Ingress controller than the Minikube one). PAWS is tra... [15:46:23] 10Quarry, 06tools-platform-team: Quarry should support connecting to alternative extension databases - https://phabricator.wikimedia.org/T395800#12295337 (10taavi) [15:47:23] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [15:47:48] 10Toolforge, 06tools-platform-team: tool registration failed AKA delete tool wikiblame and/or replace it with a copy of tool blame - https://phabricator.wikimedia.org/T437214#12295340 (10dcaro) 05Open→03Resolved a:03dcaro The tool was already created today, probably on a first attempt. [15:47:56] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [15:48:37] 10Data-Services, 06tools-platform-team, 07User-notice: Update Wiki Replicas documentation for x4 rollout - https://phabricator.wikimedia.org/T437208#12295345 (10taavi) This should go in Tech News as well. This was announced as https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thr... [15:53:03] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [15:53:59] 10Data-Services, 06tools-platform-team, 07User-notice: Update Wiki Replicas documentation for x4 rollout - https://phabricator.wikimedia.org/T437208#12295366 (10taavi) Added x4 extension aliases to the documentation as https://wikitech.wikimedia.org/w/index.php?title=Help:Wiki_Replicas&diff=prev&oldid=245509... [15:55:24] (03approved) 10dcaro: components-api: bump to 0.0.213-20260907152616-34306785 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1413 (https://phabricator.wikimedia.org/T433690) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:55:33] (03merge) 10dcaro: components-api: bump to 0.0.213-20260907152616-34306785 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1413 (https://phabricator.wikimedia.org/T433690) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:57:57] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [components-api] Create endpoint to get tools based on a set config - https://phabricator.wikimedia.org/T433690#12295390 (10dcaro) [15:58:27] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [components-api] Create endpoint to get tools based on a set config - https://phabricator.wikimedia.org/T433690#12295392 (10dcaro) Example request: ` dcaro@acme$ curl https://api.svc.toolforge.org/components/v1/config/search --json '{"matc... [15:59:27] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [components-api] Create endpoint to get tools based on a set config - https://phabricator.wikimedia.org/T433690#12295395 (10dcaro) 05In progress→03Resolved [16:24:31] (03merge) 10dcaro: tests: add some tests for runtime.build_start [repos/cloud/toolforge/components-api] (add_build_envvars) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/185 (https://phabricator.wikimedia.org/T434496) [16:24:36] (03update) 10dcaro: api: add envvars to source build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/184 (https://phabricator.wikimedia.org/T434496) [17:13:17] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/44 (owner: 10l10n-bot) [17:13:23] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/44 (owner: 10l10n-bot) [17:24:21] (03update) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [17:24:32] (03update) 10tlepage: refactor: Parse Loki result streams into events [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [17:24:56] (03update) 10tlepage: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] (write-refactor-for-read) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) [17:24:58] (03update) 10tlepage: refactor: Generalize job event models [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) [18:07:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:12:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:57:09] 10Data-Services, 06tools-platform-team, 07User-notice: Update Wiki Replicas documentation for x4 rollout - https://phabricator.wikimedia.org/T437208#12295764 (10taavi) Created https://wikitech.wikimedia.org/wiki/News/2026_Commons_links_tables_database_split to collect the instructions in an editable place si... [19:01:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:06:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:08:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:13:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:56:01] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12295799 (10Southparkfan) `deployment-mediawiki15` and `deployment-me... [20:08:49] 10Tool-wiki-gender-stats: Update Spanish IPA symbols - https://phabricator.wikimedia.org/T437251 (10Danya) 03NEW [21:17:37] (03PS1) 10Wandji collins: Auth implementation for login and updating wikidat entry [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1337661 (https://phabricator.wikimedia.org/T409648) [21:26:16] (03CR) 10Wandji collins: [V:03+2 C:03+2] Auth implementation for login and updating wikidat entry [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1337661 (https://phabricator.wikimedia.org/T409648) (owner: 10Wandji collins)