[00:03:39] !log cdobbins@cumin2002 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp4043.ulsfo.wmnet with OS trixie [00:03:56] !log brett@cumin2002 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4047.ulsfo.wmnet with OS trixie [00:04:23] !log brett@cumin2002 START - Cookbook sre.hosts.reimage for host cp4047.ulsfo.wmnet with OS trixie [00:10:48] !log cdobbins@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4040.ulsfo.wmnet with OS trixie [00:11:43] !log cdobbins@cumin2002 START - Cookbook sre.hosts.reimage for host cp4043.ulsfo.wmnet with OS trixie [00:11:53] !log cdobbins@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on cp4042.ulsfo.wmnet with reason: host reimage [00:13:07] (03PS3) 10RLazarus: mw-*: Update to Envoy 1.35.9 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1251187 (https://phabricator.wikimedia.org/T419637) [00:14:25] !log cdobbins@cumin2002 conftool action : set/pooled=yes; selector: name=cp4040.ulsfo.wmnet [reason: trixie reimaging] [00:15:31] !log cdobbins@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4042.ulsfo.wmnet with reason: host reimage [00:16:02] (03CR) 10RLazarus: [C:03+2] mw-*: Update to Envoy 1.35.9 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1251187 (https://phabricator.wikimedia.org/T419637) (owner: 10RLazarus) [00:18:19] (03Merged) 10jenkins-bot: mw-*: Update to Envoy 1.35.9 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1251187 (https://phabricator.wikimedia.org/T419637) (owner: 10RLazarus) [00:19:33] scapping the envoy update to mediawikis everywhere [00:21:40] !log rzl@deploy2002 Started scap sync-world: https://gerrit.wikimedia.org/r/1251187 T419637 [00:21:44] T419637: Upgrade Envoy to v1.35.9 - https://phabricator.wikimedia.org/T419637 [00:22:38] !log rzl@deploy2002 rzl: https://gerrit.wikimedia.org/r/1251187 T419637 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [00:23:50] !log brett@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on cp4047.ulsfo.wmnet with reason: host reimage [00:23:56] !log rzl@deploy2002 rzl: Continuing with sync [00:27:24] !log rzl@deploy2002 Finished scap sync-world: https://gerrit.wikimedia.org/r/1251187 T419637 (duration: 07m 12s) [00:27:28] T419637: Upgrade Envoy to v1.35.9 - https://phabricator.wikimedia.org/T419637 [00:30:14] !log brett@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4047.ulsfo.wmnet with reason: host reimage [00:31:35] 06SRE, 06collaboration-services, 10Wikimedia-Mailing-lists, 07Performance Issue, 07Upstream: https://lists.wikimedia.org is often slow to load - https://phabricator.wikimedia.org/T353891#11704947 (10tstarling) >>! In T353891#11704396, @bd808 wrote: > https://lists.wikimedia.org/postorius/lists/mediawiki-... [00:31:53] !log cdobbins@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on cp4043.ulsfo.wmnet with reason: host reimage [00:39:10] !log cdobbins@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4043.ulsfo.wmnet with reason: host reimage [00:39:13] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1251199 [00:39:13] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1251199 (owner: 10TrainBranchBot) [00:41:45] !log cdobbins@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4042.ulsfo.wmnet with OS trixie [00:42:01] !log cdobbins@cumin2002 conftool action : set/pooled=yes; selector: name=cp4042.ulsfo.wmnet [reason: trixie reimaging] [00:45:11] !log cdobbins@cumin2002 conftool action : set/pooled=no; selector: name=cp4044.ulsfo.wmnet [reason: trixie reimaging] [00:45:47] !log cdobbins@cumin2002 START - Cookbook sre.hosts.reimage for host cp4044.ulsfo.wmnet with OS trixie [00:51:34] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1251199 (owner: 10TrainBranchBot) [00:55:08] !log brett@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4047.ulsfo.wmnet with OS trixie [00:59:48] (03PS1) 10Gerrit Patch Uploader: ptwiki: Enable block action for the abuse filter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251200 (https://phabricator.wikimedia.org/T419312) [00:59:50] (03CR) 10Gerrit Patch Uploader: "This commit was uploaded using the Gerrit Patch Uploader [1]." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251200 (https://phabricator.wikimedia.org/T419312) (owner: 10Gerrit Patch Uploader) [01:05:03] !log cdobbins@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4043.ulsfo.wmnet with OS trixie [01:06:35] !log cdobbins@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on cp4044.ulsfo.wmnet with reason: host reimage [01:08:59] !log cdobbins@cumin2002 conftool action : set/pooled=yes; selector: name=cp4043.ulsfo.wmnet [reason: trixie reimaging] [01:09:14] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1251201 [01:09:14] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1251201 (owner: 10TrainBranchBot) [01:09:51] !log cdobbins@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4044.ulsfo.wmnet with reason: host reimage [01:14:18] (03PS3) 10Dzahn: jenkins: add ci::httpd profile to role [puppet] - 10https://gerrit.wikimedia.org/r/1250752 (https://phabricator.wikimedia.org/T418521) [01:17:11] (03PS4) 10Dzahn: jenkins: add ci::httpd profile to role [puppet] - 10https://gerrit.wikimedia.org/r/1250752 (https://phabricator.wikimedia.org/T418521) [01:17:55] (03CR) 10Dzahn: [C:03+2] "unblocked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1250752" [puppet] - 10https://gerrit.wikimedia.org/r/1250755 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [01:18:14] (03CR) 10Dzahn: [V:03+1 C:03+1] "https://puppet-compiler.wmflabs.org/output/1250752/8263/contint2003.wikimedia.org/index.html" [puppet] - 10https://gerrit.wikimedia.org/r/1250752 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [01:18:39] (03CR) 10Dzahn: [V:03+1 C:03+2] jenkins: add ci::httpd profile to role [puppet] - 10https://gerrit.wikimedia.org/r/1250752 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [01:22:16] !log brett@puppetserver1001 conftool action : set/pooled=yes; selector: name=cp4047.* [01:22:40] !log dzahn@cumin2002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on contint1003.wikimedia.org with reason: setup [01:23:23] !log dzahn@cumin2002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on contint2003.wikimedia.org with reason: setup [01:24:27] (03CR) 10Dzahn: [V:03+1 C:03+2] "noop on contint prod - setup in progress on new jenkins" [puppet] - 10https://gerrit.wikimedia.org/r/1250752 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [01:26:39] !log dzahn@cumin2002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on contint2003.wikimedia.org with reason: T418521 [01:26:43] T418521: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521 [01:26:53] !log dzahn@cumin2002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on contint1003.wikimedia.org with reason: T418521 [01:27:50] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1251201 (owner: 10TrainBranchBot) [01:37:03] (03PS1) 10RLazarus: mw-videoscaler: Update to Envoy 1.35.9 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1251203 (https://phabricator.wikimedia.org/T419637) [01:37:20] !log contint1003/contint2003 - every time(?) we setup machines with puppet using our httpd module and PHP - and puppet runs for the first time we run into the same old issue with "Exec[ensure_present_mod_php" failing and "Considering conflict mpm_worker for mpm_prefork"sudo a2dismod mpm_event". The fix is: 'sudo a2dismod mpm_event' and run puppet again. T418521 [01:37:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:37:24] T418521: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521 [01:38:18] (03PS1) 10RLazarus: kubernetes: Set default Envoy version to 1.35.9 [puppet] - 10https://gerrit.wikimedia.org/r/1251204 (https://phabricator.wikimedia.org/T419637) [01:41:27] !log cdobbins@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4044.ulsfo.wmnet with OS trixie [01:43:26] 10ops-eqsin, 06SRE, 06DC-Ops, 06Traffic: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#11704980 (10RobH) Tech is running late, their dispatcher called me to let me know. They were set to be onsite at 7AM, but it will now be closer to 10:30AM / 19:30 Pacific [01:45:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:48:30] (03PS2) 10RLazarus: mw-parsoid: Delete values-canary.yaml and values-migration.yaml [deployment-charts] - 10https://gerrit.wikimedia.org/r/1251192 (https://phabricator.wikimedia.org/T386246) [01:48:38] (03PS3) 10Dzahn: jenkins: add proxy_jenkins profile to role [puppet] - 10https://gerrit.wikimedia.org/r/1250748 (https://phabricator.wikimedia.org/T418521) [01:48:56] (03CR) 10Scott French: mw-videoscaler: Update to Envoy 1.35.9 (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1251203 (https://phabricator.wikimedia.org/T419637) (owner: 10RLazarus) [01:50:10] (03PS2) 10RLazarus: mw-videoscaler: Update to Envoy 1.35.9 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1251203 (https://phabricator.wikimedia.org/T419637) [01:50:19] (03CR) 10Scott French: [C:03+1] kubernetes: Set default Envoy version to 1.35.9 [puppet] - 10https://gerrit.wikimedia.org/r/1251204 (https://phabricator.wikimedia.org/T419637) (owner: 10RLazarus) [01:50:49] (03CR) 10Dzahn: [V:03+1 C:03+1] "https://puppet-compiler.wmflabs.org/output/1250748/8264/contint1003.wikimedia.org/index.html" [puppet] - 10https://gerrit.wikimedia.org/r/1250748 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [01:51:17] (03CR) 10RLazarus: mw-videoscaler: Update to Envoy 1.35.9 (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1251203 (https://phabricator.wikimedia.org/T419637) (owner: 10RLazarus) [01:53:40] (03CR) 10Scott French: [C:03+1] mw-videoscaler: Update to Envoy 1.35.9 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1251203 (https://phabricator.wikimedia.org/T419637) (owner: 10RLazarus) [01:54:52] (03CR) 10Dzahn: [V:03+1 C:03+1] "merging this will enable the /ci URL on the domain we host on this and proxy to jenkins. jenkins.discovery.wmnet/ci/ ?" [puppet] - 10https://gerrit.wikimedia.org/r/1250748 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [01:56:24] (03PS4) 10Dzahn: jenkins: add proxy_jenkins profile to role [puppet] - 10https://gerrit.wikimedia.org/r/1250748 (https://phabricator.wikimedia.org/T418521) [02:00:47] !log mwpresync@deploy2002 Started scap build-images: Publishing wmf/next image [02:01:27] (03CR) 10Dzahn: "what this adds can be seen in puppet repo (puppet:///modules/contint/apache/proxy_jenkins) or existing contint1002/2002 with "cat /etc/ap" [puppet] - 10https://gerrit.wikimedia.org/r/1250748 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [02:04:59] FIRING: CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [02:08:39] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:09:06] !log mwpresync@deploy2002 Finished scap build-images: Publishing wmf/next image (duration: 08m 18s) [02:14:59] FIRING: [3x] HelmReleaseBadStatus: Helm release kserve/kserve on k8s-mlserve@codfw in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [02:15:42] (03PS1) 10Dzahn: jenkins: add envoy and config for jenkins.discovery.wmnet (WIP) [puppet] - 10https://gerrit.wikimedia.org/r/1251205 (https://phabricator.wikimedia.org/T418521) [02:23:29] 06SRE, 10envoy, 06ServiceOps new, 10ServiceOps-Services-Oids: Upgrade Envoy to v1.35.7 - https://phabricator.wikimedia.org/T410975#11704988 (10RLazarus) 05In progress→03Resolved Resolving; the remaining hosts will go straight to 1.35.9 in T419637 instead. >>! In T410975#11681081, @MLechvien-WMF wr... [02:33:39] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:51:55] 10ops-eqsin, 06SRE, 06DC-Ops, 06Traffic: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#11705018 (10RobH) Tech is onsite and performing the hw power distro board swap on cp5022 [02:57:03] !log cdobbins@cumin2002 conftool action : set/pooled=yes; selector: name=cp4044.ulsfo.wmnet [reason: trixie reimaging] [02:59:59] FIRING: [2x] KubernetesCalicoDown: dse-k8s-worker1018.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [03:07:23] (03CR) 10ArielGlenn: "Tiny typo, otherwise looks good." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1251148 (owner: 10Daniel Kinzler) [03:16:14] (03PS1) 10Dzahn: jenkins: enable the jenkins service if using new role [puppet] - 10https://gerrit.wikimedia.org/r/1251208 (https://phabricator.wikimedia.org/T418521) [03:16:49] (03CR) 10CI reject: [V:04-1] jenkins: enable the jenkins service if using new role [puppet] - 10https://gerrit.wikimedia.org/r/1251208 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [03:23:11] (03PS2) 10Dzahn: jenkins: enable the jenkins service if using new role [puppet] - 10https://gerrit.wikimedia.org/r/1251208 (https://phabricator.wikimedia.org/T418521) [03:28:47] (03PS3) 10Dzahn: jenkins: enable the jenkins service if using new role [puppet] - 10https://gerrit.wikimedia.org/r/1251208 (https://phabricator.wikimedia.org/T418521) [03:32:03] (03CR) 10Dzahn: [V:03+1 C:03+1] "https://puppet-compiler.wmflabs.org/output/1251208/8267/" [puppet] - 10https://gerrit.wikimedia.org/r/1251208 (https://phabricator.wikimedia.org/T418521) (owner: 10Dzahn) [03:55:16] (03CR) 10ArielGlenn: "Except for Claime's changes related to the spec endpoint, it makes sense to me and looks right. I did not attempt to test this however." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1251100 (owner: 10Clément Goubert) [04:26:19] PROBLEM - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2002 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [04:27:51] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for SCardenas (WMF) - https://phabricator.wikimedia.org/T419932 (10Scardenasmolinar) 03NEW [05:26:19] RECOVERY - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2002 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [05:45:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:48:13] (03CR) 10Dreamrimmer: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251193 (https://phabricator.wikimedia.org/T419105) (owner: 10Codename Noreste) [05:57:14] (03CR) 10Dreamrimmer: idwiki: Remove unused user groups on Indonesian Wikipedia (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251193 (https://phabricator.wikimedia.org/T419105) (owner: 10Codename Noreste) [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260313T0600) [06:04:59] FIRING: CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [06:14:58] (03CR) 10Anzx: [C:03+1] "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251200 (https://phabricator.wikimedia.org/T419312) (owner: 10Gerrit Patch Uploader) [06:14:59] FIRING: [3x] HelmReleaseBadStatus: Helm release kserve/kserve on k8s-mlserve@codfw in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [06:15:45] (03CR) 10CI reject: [V:04-1] ptwiki: Enable block action for the abuse filter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251200 (https://phabricator.wikimedia.org/T419312) (owner: 10Gerrit Patch Uploader) [06:17:28] (03CR) 10Anzx: ptwiki: Enable block action for the abuse filter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251200 (https://phabricator.wikimedia.org/T419312) (owner: 10Gerrit Patch Uploader) [06:38:11] (03CR) 10Anzx: ptwiki: Enable block action for the abuse filter (034 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1251200 (https://phabricator.wikimedia.org/T419312) (owner: 10Gerrit Patch Uploader) [06:59:59] FIRING: [2x] KubernetesCalicoDown: dse-k8s-worker1018.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260313T0700) [07:19:51] (03CR) 10Arnaudb: [C:03+2] mailman: update the web frontend firewall rules [puppet] - 10https://gerrit.wikimedia.org/r/1251009 (https://phabricator.wikimedia.org/T286066) (owner: 10Arnaudb) [07:23:51] 06SRE, 06collaboration-services, 10Wikimedia-Mailing-lists, 13Patch-For-Review: Put lists.wikimedia.org web interface behind CDN - https://phabricator.wikimedia.org/T286066#11705274 (10ABran-WMF) 05In progress→03Resolved mailman-web is now fully behind CDN: ` ~ $ curl -s https://lists1004.wikimedi... [07:54:17] (03PS1) 10Elukey: kserve: Remove caBundle occurrences [deployment-charts] - 10https://gerrit.wikimedia.org/r/1251220 (https://phabricator.wikimedia.org/T419040) [07:55:50] !log installing 6.12.74 on Trixie hosts [07:55:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:56:41] !log installing Linux 6.12.74 on Trixie hosts [07:56:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:17:47] 06SRE, 06collaboration-services, 06Infrastructure-Foundations, 10Mail, and 2 others: Replace Spamassassin with Rspam for VRTS on Postfix - https://phabricator.wikimedia.org/T402260#11705361 (10ABran-WMF) 05Stalled→03In progress [08:37:33] !log elukey@cumin1003 START - Cookbook sre.kafka.change-confluent-distro-version Change Confluent distribution for Kafka A:kafka-test-eqiad cluster: Change Confluent distribution. [08:41:35] elukey@cumin1003 change-confluent-distro-version (PID 3160738) is awaiting input [08:41:57] (03CR) 10Elukey: [C:03+2] role::kafka::test::broker: move to Confluent Kafka 3.7 [puppet] - 10https://gerrit.wikimedia.org/r/1249940 (https://phabricator.wikimedia.org/T417035) (owner: 10Elukey) [08:46:56] (03CR) 10Silvan Heintze: [C:03+1] "LGTM, tried it out locally - works nicely 👍" [dumps] - 10https://gerrit.wikimedia.org/r/1251169 (https://phabricator.wikimedia.org/T401296) (owner: 10WMDE-leszek) [08:52:25] (03PS10) 10Jcrespo: mediabackups: Initial puppetization of Versity S3 gateway for replacing minio [puppet] - 10https://gerrit.wikimedia.org/r/1251113 (https://phabricator.wikimedia.org/T410020) [08:52:47] (03CR) 10Jcrespo: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1251113 (https://phabricator.wikimedia.org/T410020) (owner: 10Jcrespo)