[00:54:53] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr2-esams:xe-0/1/7 (Transit: Liberty Global (BB00088)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-esams:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [00:59:39] RESOLVED: [2x] TransitBGPDown: Transit BGP session down between cr2-esams and LibertyGlobal (2001:730:2209:1::d52e:ba09) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [01:10:52] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1340734 [01:10:52] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1340734 (owner: 10TrainBranchBot) [01:20:01] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1340734 (owner: 10TrainBranchBot) [02:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260913T0700) [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260914T0200) [02:00:31] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:06:40] FIRING: SystemdUnitFailed: production-images-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:40:11] 06SRE, 06Traffic, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12315632 (10MauricioGenta) I understand @Nemoralis even if a disagree, i started the thread because a user called my attention telling me... [02:52:02] (03PS1) 10Tim Starling: Allow title-like strings with Package: prefix in require() [extensions/Scribunto] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1340801 (https://phabricator.wikimedia.org/T430644) [02:52:15] (03PS1) 10Tim Starling: Runtime: Add a facility for loading files by title [extensions/Produnto] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1340802 (https://phabricator.wikimedia.org/T430644) [02:52:22] (03CR) 10CI reject: [V:04-1] Allow title-like strings with Package: prefix in require() [extensions/Scribunto] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1340801 (https://phabricator.wikimedia.org/T430644) (owner: 10Tim Starling) [02:53:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by tstarling@deploy1003 using scap backport" [extensions/Scribunto] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1340801 (https://phabricator.wikimedia.org/T430644) (owner: 10Tim Starling) [02:53:19] (03CR) 10TrainBranchBot: [C:03+2] "Approved by tstarling@deploy1003 using scap backport" [extensions/Produnto] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1340802 (https://phabricator.wikimedia.org/T430644) (owner: 10Tim Starling) [02:57:07] (03Merged) 10jenkins-bot: Runtime: Add a facility for loading files by title [extensions/Produnto] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1340802 (https://phabricator.wikimedia.org/T430644) (owner: 10Tim Starling) [02:58:34] (03Merged) 10jenkins-bot: Allow title-like strings with Package: prefix in require() [extensions/Scribunto] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1340801 (https://phabricator.wikimedia.org/T430644) (owner: 10Tim Starling) [02:59:50] !log tstarling@deploy1003 Started scap sync-world: Backport for [[gerrit:1340801|Allow title-like strings with Package: prefix in require() (T430644)]], [[gerrit:1340802|Runtime: Add a facility for loading files by title (T430644)]] [02:59:54] T430644: Re-consider module paths for access to Produnto - https://phabricator.wikimedia.org/T430644 [03:00:34] !log tstarling@deploy1003 sync-world failed: Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py --http-proxy http://webproxy:8080 --https-proxy http://webproxy:8080 /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.47.0-wmf.18,1.47.0-wmf.19,next --multiversion-image-basename docker-registry.discovery.wmnet/restricted [03:00:34] /mediawiki-multiversion --singleversion-image-basename docker-registry.discovery.wmnet/restricted/mediawiki-singleversion --webserver-image-name docker-registry.discovery.wmnet/restricted/mediawiki-webserver --latest-tag latest --label vnd.wikimedia.builder.name=scap --label vnd.wikimedia.builder.version=4.289.0 --label vnd.wikimedia.scap.stage_dir=/srv/mediawiki-staging --label vnd.wikimedia.scap.build_state_dir=/srv/med [03:00:34] iawiki-staging/scap/image-build' returned non-zero exit status 1. (scap version: 4.289.0) (duration: 00m 44s) [03:01:48] fun [03:04:20] 03:00:32 [mediawiki-publish-83] E: Release file for http://snapshot.debian.org/archive/debian/20260906T040200Z/dists/bookworm-updates/InRelease is expired (invalid since 1d 0h 47min 46s). Updates for this repository will not be applied. [03:04:20] 03:00:32 [mediawiki-publish-83] E: Release file for http://snapshot.debian.org/archive/debian-security/20260906T040200Z/dists/bookworm-security/InRelease is expired (invalid since 23h 36min 5s). Updates for this repository will not be applied. [03:04:34] FIRING: CertAlmostExpired: Certificate for service contint1002:1443 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#contint1002:1443 - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [03:04:35] (03PS1) 10Pppery: Add new namespace for zh_min_nanwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340815 (https://phabricator.wikimedia.org/T430731) [03:05:09] (03PS2) 10Pppery: Add new namespace for zh_min_nanwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340815 (https://phabricator.wikimedia.org/T430731) [03:14:03] 06SRE, 06Traffic, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12315670 (10Pppery) This "decision" was taken in the code review of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1069643, where @D... [03:35:39] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:45:37] 06SRE, 06Traffic, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12315691 (10MauricioGenta) Oh cool, I didn't think we had so many domains! I trust your judgment would be the better, i do not want to bre... [03:49:37] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [04:31:37] (03CR) 10Ryan Kemper: sre.hadoop.roll-restart-workers: optionally skip hosts (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1339776 (https://phabricator.wikimedia.org/T437713) (owner: 10Bking) [04:32:43] (03CR) 10Ryan Kemper: [C:03+2] "specifically it'd look something like (sorry gerrit is clobbering some whitespace):" [cookbooks] - 10https://gerrit.wikimedia.org/r/1214664 (https://phabricator.wikimedia.org/T411568) (owner: 10Ryan Kemper) [04:50:17] 06SRE, 06Traffic, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12315775 (10Joe) My 2 cents: I think the redirect should be changed. No one expects a site with an argentinian TLD to be in arabic. [04:52:58] !log on build2001 rebuilding base images T437829 [04:53:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:53:02] T437829: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829 [05:39:57] <_joe_> !log force-running build-base-images on build2004 for T437829 [05:39:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:40:00] T437829: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829 [05:47:11] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 14 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1330446 (https://phabricator.wikimedia.org/T419684) (owner: 10Arendpieter) [05:51:27] <_joe_> !log pulled bookworm:latest from build2004 to build2001 T437829 [05:51:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:51:30] T437829: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829 [06:05:38] (03PS1) 10Giuseppe Lavagetto: docker::registry: fix location of hiera key [puppet] - 10https://gerrit.wikimedia.org/r/1340942 [06:05:39] (03PS1) 10Giuseppe Lavagetto: build-production-images: remove buster from base images [puppet] - 10https://gerrit.wikimedia.org/r/1340943 (https://phabricator.wikimedia.org/T437829) [06:06:23] 06SRE, 10Wikimedia-Mailing-lists: create a new mailing list for ar-arbcom - https://phabricator.wikimedia.org/T437832#12315819 (10HideonRosie) [06:06:40] FIRING: SystemdUnitFailed: production-images-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:07:11] (03CR) 10Giuseppe Lavagetto: [V:03+1] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9411/console" [puppet] - 10https://gerrit.wikimedia.org/r/1340942 (owner: 10Giuseppe Lavagetto) [06:07:29] (03CR) 10Giuseppe Lavagetto: [V:03+1 C:03+2] docker::registry: fix location of hiera key [puppet] - 10https://gerrit.wikimedia.org/r/1340942 (owner: 10Giuseppe Lavagetto) [06:09:28] (03CR) 10Giuseppe Lavagetto: [C:03+2] build-production-images: remove buster from base images [puppet] - 10https://gerrit.wikimedia.org/r/1340943 (https://phabricator.wikimedia.org/T437829) (owner: 10Giuseppe Lavagetto) [06:16:46] (03PS1) 10Giuseppe Lavagetto: php: remove stale images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1340948 [06:26:34] !log on deploy1003: docker image pull docker-registry.wikimedia.org/php8.3-fpm-multiversion-base [06:26:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:39:20] RESOLVED: CertAlmostExpired: Certificate for service contint1002:1443 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#contint1002:1443 - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [06:54:11] 06SRE, 06Infrastructure-Foundations, 06Release-Engineering-Team (Radar): apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12315884 (10MoritzMuehlenhoff) >>! In T437069#12304969, @Scott_French wrote: > @MoritzMuehlenhoff - Do you happen to have any... [06:56:53] !log tstarling@deploy1003 Started scap sync-world: Backport for [[gerrit:1340801|Allow title-like strings with Package: prefix in require() (T430644)]], [[gerrit:1340802|Runtime: Add a facility for loading files by title (T430644)]] [06:56:57] T430644: Re-consider module paths for access to Produnto - https://phabricator.wikimedia.org/T430644 [06:58:40] (03PS1) 10Muehlenhoff: Disable bullseye-security in apt config for Bullseye [puppet] - 10https://gerrit.wikimedia.org/r/1340987 (https://phabricator.wikimedia.org/T437069) [07:00:04] Amir1, urbanecm, and awight: Time to do the UTC morning backport window deploy. Don't look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260914T0700). [07:00:04] No Gerrit patches in the queue for this window AFAICS. [07:01:49] (03CR) 10Giuseppe Lavagetto: [V:03+1] "PCC SUCCESS (DIFF 8): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9412/console" [puppet] - 10https://gerrit.wikimedia.org/r/1330130 (owner: 10Giuseppe Lavagetto) [07:04:50] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1340987 (https://phabricator.wikimedia.org/T437069) (owner: 10Muehlenhoff) [07:14:15] FIRING: [2x] ProbeDown: Service idp1005:443 has failed probes (http_idp_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/CAS-SSO#Alerting - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:14:50] (03CR) 10Jelto: "two comments in line" [cookbooks] - 10https://gerrit.wikimedia.org/r/1339602 (https://phabricator.wikimedia.org/T437680) (owner: 10Arnaudb) [07:15:15] (03CR) 10Giuseppe Lavagetto: [V:03+1 C:03+2] containerd: use hiera parameters to detect if dragonfly is enabled [puppet] - 10https://gerrit.wikimedia.org/r/1330130 (owner: 10Giuseppe Lavagetto) [07:16:29] (03PS1) 10Muehlenhoff: Record LDAP access for vcioffi [puppet] - 10https://gerrit.wikimedia.org/r/1340993 [07:17:34] (03CR) 10Giuseppe Lavagetto: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9413/co" [puppet] - 10https://gerrit.wikimedia.org/r/1330129 (https://phabricator.wikimedia.org/T435796) (owner: 10Giuseppe Lavagetto) [07:17:46] !log tstarling@deploy1003 tstarling: Backport for [[gerrit:1340801|Allow title-like strings with Package: prefix in require() (T430644)]], [[gerrit:1340802|Runtime: Add a facility for loading files by title (T430644)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:17:49] T430644: Re-consider module paths for access to Produnto - https://phabricator.wikimedia.org/T430644 [07:18:04] (03CR) 10Muehlenhoff: [C:03+2] Record LDAP access for vcioffi [puppet] - 10https://gerrit.wikimedia.org/r/1340993 (owner: 10Muehlenhoff) [07:18:31] !log tstarling@deploy1003 tstarling: Continuing with deployment [07:19:14] (03CR) 10Giuseppe Lavagetto: [V:03+1 C:03+2] profile::containerd: add gVisor support [puppet] - 10https://gerrit.wikimedia.org/r/1330129 (https://phabricator.wikimedia.org/T435796) (owner: 10Giuseppe Lavagetto) [07:19:15] RESOLVED: [2x] ProbeDown: Service idp1005:443 has failed probes (http_idp_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/CAS-SSO#Alerting - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:19:22] (03PS7) 10Giuseppe Lavagetto: profile::containerd: add gVisor support [puppet] - 10https://gerrit.wikimedia.org/r/1330129 (https://phabricator.wikimedia.org/T435796) [07:22:01] (03CR) 10Giuseppe Lavagetto: [C:03+2] profile::containerd: add gVisor support [puppet] - 10https://gerrit.wikimedia.org/r/1330129 (https://phabricator.wikimedia.org/T435796) (owner: 10Giuseppe Lavagetto) [07:24:58] (03CR) 10Muehlenhoff: [C:03+1] "Looks good, can be merged once group approval is in" [puppet] - 10https://gerrit.wikimedia.org/r/1339906 (https://phabricator.wikimedia.org/T437658) (owner: 10Ssingh) [07:25:05] (03CR) 10Jelto: [C:03+2] fix helm command in all NOTES.txt [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333699 (https://phabricator.wikimedia.org/T433589) (owner: 10Jelto) [07:29:20] (03PS3) 10Arnaudb: gitlab: offer to skip backup to the operator for replicas [cookbooks] - 10https://gerrit.wikimedia.org/r/1339602 (https://phabricator.wikimedia.org/T437680) [07:31:23] !log tstarling@deploy1003 Finished scap sync-world: Backport for [[gerrit:1340801|Allow title-like strings with Package: prefix in require() (T430644)]], [[gerrit:1340802|Runtime: Add a facility for loading files by title (T430644)]] (duration: 34m 30s) [07:31:28] T430644: Re-consider module paths for access to Produnto - https://phabricator.wikimedia.org/T430644 [07:32:11] (03CR) 10Giuseppe Lavagetto: [V:03+1 C:03+2] kubernetes::staging: make containerd support gVisor runtime in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1330131 (https://phabricator.wikimedia.org/T435796) (owner: 10Giuseppe Lavagetto) [07:32:35] 06SRE, 10SRE-Access-Requests: Requesting access to production (ops group) for dkertesz - https://phabricator.wikimedia.org/T437835 (10dkertesz) 03NEW [07:35:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:35:49] (03PS1) 10Daniel Kertesz: admin: move dkertesz from ops-limited to ops [puppet] - 10https://gerrit.wikimedia.org/r/1341006 (https://phabricator.wikimedia.org/T437835) [07:43:01] (03Merged) 10jenkins-bot: fix helm command in all NOTES.txt [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333699 (https://phabricator.wikimedia.org/T433589) (owner: 10Jelto) [07:44:22] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply [07:45:09] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply [07:45:10] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply [07:45:47] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply [07:45:48] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply [07:47:06] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply [07:47:07] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply [07:47:20] (03PS1) 10Volans: transports.clustershell: report also empty outputs [software/cumin] - 10https://gerrit.wikimedia.org/r/1341027 [07:47:21] (03PS1) 10Volans: transports: add status to the CommandResults class [software/cumin] - 10https://gerrit.wikimedia.org/r/1341028 [07:47:21] (03PS1) 10Volans: puppetdb backend: JSON-encode query interpolations [software/cumin] - 10https://gerrit.wikimedia.org/r/1341029 [07:47:21] (03PS1) 10Volans: query: detect circular alias references [software/cumin] - 10https://gerrit.wikimedia.org/r/1341030 [07:47:22] (03PS1) 10Volans: puppetdb backend: reset endpoint on each build [software/cumin] - 10https://gerrit.wikimedia.org/r/1341031 [07:47:24] (03PS1) 10Volans: backends: use pyparsing PEP8 names [software/cumin] - 10https://gerrit.wikimedia.org/r/1341032 [07:47:28] (03PS1) 10Volans: transports: disable Tqdm monitor thread [software/cumin] - 10https://gerrit.wikimedia.org/r/1341033 [07:47:32] (03PS1) 10Volans: tests: disable deprecation warning [software/cumin] - 10https://gerrit.wikimedia.org/r/1341034 [07:47:53] 06SRE, 06Collaboration-Services, 10GitLab: "apt update" failing in CI - https://phabricator.wikimedia.org/T437836 (10MatthewVernon) 03NEW [07:47:55] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply [07:47:57] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply [07:48:13] (03CR) 10Arnaudb: gitlab: offer to skip backup to the operator for replicas (032 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1339602 (https://phabricator.wikimedia.org/T437680) (owner: 10Arnaudb) [07:48:15] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply [07:48:17] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply [07:48:23] 06SRE, 06Collaboration-Services, 10GitLab: "apt update" failing in CI - https://phabricator.wikimedia.org/T437836#12315981 (10MatthewVernon) [07:48:35] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply [07:48:36] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [07:48:38] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [07:48:39] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply [07:49:01] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply [07:49:02] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply [07:49:34] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply [07:49:35] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [07:49:37] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [07:50:07] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [07:50:07] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply [07:50:21] (03PS2) 10Muehlenhoff: Stop the build of Bullseye PHP base images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1337932 (https://phabricator.wikimedia.org/T417389) [07:50:46] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply [07:50:47] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply [07:51:16] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply [07:51:17] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply [07:51:47] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply [07:51:48] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply [07:52:21] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply [07:52:22] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply [07:53:19] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply [07:53:20] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply [07:54:04] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply [07:54:05] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply [07:54:36] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply [07:54:37] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply [07:55:04] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply [07:55:05] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [07:55:43] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [07:55:44] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply [07:56:18] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply [07:56:19] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply [07:56:28] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Stop the build of Bullseye PHP base images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1337932 (https://phabricator.wikimedia.org/T417389) (owner: 10Muehlenhoff) [07:56:59] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply [07:57:53] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [07:58:16] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 00m 23s) [08:04:36] (03PS4) 10Arnaudb: gitlab: offer to skip backup to the operator for replicas [cookbooks] - 10https://gerrit.wikimedia.org/r/1339602 (https://phabricator.wikimedia.org/T437680) [08:07:53] (03CR) 10Slyngshede: [C:03+2] P:tofurkey enable Tofurkey for MAGRU [puppet] - 10https://gerrit.wikimedia.org/r/1324689 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [08:08:22] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [08:10:07] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [08:11:00] (03PS1) 10Arnaudb: gitlab.wm.o: allow library-default UAs [puppet] - 10https://gerrit.wikimedia.org/r/1338946 (https://phabricator.wikimedia.org/T425441) [08:16:54] (03PS2) 10Majavah: hieradata: cloudlb: Remove temporary x4 section_port override [puppet] - 10https://gerrit.wikimedia.org/r/1338949 (https://phabricator.wikimedia.org/T437208) [08:17:50] (03PS1) 10Santiago Faci: Test Kitchen UI: Deploy v1.5.5 release to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341040 (https://phabricator.wikimedia.org/T429524) [08:18:22] (03PS1) 10Majavah: P:wmcs: maintain-views: Set x4 database list [puppet] - 10https://gerrit.wikimedia.org/r/1341041 (https://phabricator.wikimedia.org/T431309) [08:18:24] (03PS2) 10Santiago Faci: Test Kitchen UI: Deploy v1.5.5 release to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341040 (https://phabricator.wikimedia.org/T429524) [08:19:33] (03PS1) 10Marostegui: dbstore1007: Add x4 [puppet] - 10https://gerrit.wikimedia.org/r/1341042 (https://phabricator.wikimedia.org/T437839) [08:22:03] (03PS2) 10Muehlenhoff: Remove python-build Bullseye image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1339635 (https://phabricator.wikimedia.org/T416452) [08:22:55] (03CR) 10FNegri: [C:03+1] P:wmcs: maintain-views: Set x4 database list [puppet] - 10https://gerrit.wikimedia.org/r/1341041 (https://phabricator.wikimedia.org/T431309) (owner: 10Majavah) [08:23:16] (03CR) 10Majavah: [C:03+2] P:wmcs: maintain-views: Set x4 database list [puppet] - 10https://gerrit.wikimedia.org/r/1341041 (https://phabricator.wikimedia.org/T431309) (owner: 10Majavah) [08:30:59] (03CR) 10Jelto: [C:03+1] "lgtm now, thank you" [cookbooks] - 10https://gerrit.wikimedia.org/r/1339602 (https://phabricator.wikimedia.org/T437680) (owner: 10Arnaudb) [08:31:16] (03CR) 10Arnaudb: [C:03+2] gitlab: offer to skip backup to the operator for replicas [cookbooks] - 10https://gerrit.wikimedia.org/r/1339602 (https://phabricator.wikimedia.org/T437680) (owner: 10Arnaudb) [08:31:38] (03CR) 10Hashar: [C:03+1] "For the legacy Zuul, we still used a Bullseye image with Python 2.7 (`py2-bullseye`). since we were the sole remaining use case, it got r" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1339635 (https://phabricator.wikimedia.org/T416452) (owner: 10Muehlenhoff) [08:33:36] (03CR) 10FNegri: [C:03+1] hieradata: cloudlb: Remove temporary x4 section_port override [puppet] - 10https://gerrit.wikimedia.org/r/1338949 (https://phabricator.wikimedia.org/T437208) (owner: 10Majavah) [08:33:59] (03CR) 10Phuedx: [C:03+1] Test Kitchen UI: Deploy v1.5.5 release to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341040 (https://phabricator.wikimedia.org/T429524) (owner: 10Santiago Faci) [08:34:41] (03CR) 10Majavah: [C:03+2] hieradata: cloudlb: Remove temporary x4 section_port override [puppet] - 10https://gerrit.wikimedia.org/r/1338949 (https://phabricator.wikimedia.org/T437208) (owner: 10Majavah) [08:35:08] (03Merged) 10jenkins-bot: gitlab: offer to skip backup to the operator for replicas [cookbooks] - 10https://gerrit.wikimedia.org/r/1339602 (https://phabricator.wikimedia.org/T437680) (owner: 10Arnaudb) [08:35:49] (03CR) 10Marostegui: [C:03+2] dbstore1007: Add x4 [puppet] - 10https://gerrit.wikimedia.org/r/1341042 (https://phabricator.wikimedia.org/T437839) (owner: 10Marostegui) [08:35:57] !log pruned obsolete Bullseye images php8.1-cli/php8.1-fpm/ php8.1-fpm-multiversion-base from the docker registry T416452 [08:35:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:36:00] T416452: Migrate Docker images running in Production away from Bullseye - https://phabricator.wikimedia.org/T416452 [08:36:25] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Adding x4 [08:37:23] !log taavi@cumin1003 conftool action : set/pooled=yes; selector: name=clouddb1024.eqiad.wmnet [08:37:32] !log taavi@cumin1003 conftool action : set/pooled=no; selector: name=clouddb1025.eqiad.wmnet [08:38:20] (03PS1) 10Majavah: Revert "hieradata: Route x4 traffic to the s4 replicas for now" [puppet] - 10https://gerrit.wikimedia.org/r/1341057 [08:39:06] (03CR) 10FNegri: [C:03+1] Revert "hieradata: Route x4 traffic to the s4 replicas for now" [puppet] - 10https://gerrit.wikimedia.org/r/1341057 (owner: 10Majavah) [08:39:11] 06SRE, 10Wikimedia-Mailing-lists: create a new mailing list for ar-arbcom - https://phabricator.wikimedia.org/T437832#12316161 (10Ladsgroup) Based on https://meta.wikimedia.org/wiki/Mailing_lists/Standardization the name of mailing list will be wikipedia-ar-arbcom [08:41:08] (03CR) 10Majavah: [C:03+2] Revert "hieradata: Route x4 traffic to the s4 replicas for now" [puppet] - 10https://gerrit.wikimedia.org/r/1341057 (owner: 10Majavah) [08:41:30] !log pruned obsolete Bullseye images php8.3-icu72-cli / php8.3-icu72-fpm-multiversion-base / php8.3-icu72-fpm from the docker registry T416452 [08:41:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:41:33] T416452: Migrate Docker images running in Production away from Bullseye - https://phabricator.wikimedia.org/T416452 [08:42:05] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [08:42:16] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] "php8.1-cli, php8.1-fpm, php8.1-fpm-multiversion-base, php8.3-icu72-cli, php8.3-icu72-fpm-multiversion-base, php8.3-icu72-fpm were removed " [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1337932 (https://phabricator.wikimedia.org/T417389) (owner: 10Muehlenhoff) [08:43:05] 06SRE, 06Data-Platform-SRE, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Migrate Docker images running in Production away from Bullseye - https://phabricator.wikimedia.org/T416452#12316231 (10MoritzMuehlenhoff) [08:44:24] 06SRE, 06Data-Platform-SRE, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Migrate Docker images running in Production away from Bullseye - https://phabricator.wikimedia.org/T416452#12316245 (10MoritzMuehlenhoff) All Bullseye images are gone from the production Kubernetes clusters, the only re... [08:44:25] (03CR) 10Elukey: "Not needed since Giuseppe already fixed it https://gerrit.wikimedia.org/r/c/operations/puppet/+/1340942" [puppet] - 10https://gerrit.wikimedia.org/r/1340506 (owner: 10Elukey) [08:44:32] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [08:44:32] (03Abandoned) 10Elukey: Add the docker_registry_endpoint global variable [puppet] - 10https://gerrit.wikimedia.org/r/1340506 (owner: 10Elukey) [08:50:03] (03PS3) 10Muehlenhoff: Remove python-build Bullseye image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1339635 (https://phabricator.wikimedia.org/T416452) [08:50:03] (03PS2) 10Muehlenhoff: Remove python-devel container image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1339636 (https://phabricator.wikimedia.org/T416452) [08:51:21] (03PS3) 10Muehlenhoff: Enable dbbackups::transfer for cumin1004 [puppet] - 10https://gerrit.wikimedia.org/r/1339778 (https://phabricator.wikimedia.org/T427897) [08:53:54] (03PS1) 10Majavah: P:wmcs: maintain_dbusers: Fix harvest-replicas variable shadowing [puppet] - 10https://gerrit.wikimedia.org/r/1341061 [08:55:03] (03CR) 10Jelto: [C:03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/1338134 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [09:06:00] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1024-1025].eqiad.wmnet,db[1155,1260].eqiad.wmnet,dbstore1007.eqiad.wmnet with reason: Adding x4 [09:06:16] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1260: Needs to clone another host from this one [09:06:40] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1260: Needs to clone another host from this one [09:06:58] (03CR) 10FNegri: [C:03+1] P:wmcs: maintain_dbusers: Fix harvest-replicas variable shadowing [puppet] - 10https://gerrit.wikimedia.org/r/1341061 (owner: 10Majavah) [09:07:05] !log taavi@cumin1003 conftool action : set/pooled=yes; selector: name=clouddb1025.eqiad.wmnet [09:07:46] (03CR) 10Majavah: [C:03+2] P:wmcs: maintain_dbusers: Fix harvest-replicas variable shadowing [puppet] - 10https://gerrit.wikimedia.org/r/1341061 (owner: 10Majavah) [09:08:42] !log Stop mariadb on db1260 to clone dbstore1007, there will be lag on wikireplicas:x4 https://phabricator.wikimedia.org/T437839 [09:08:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:08:45] (03CR) 10Hashar: "Back in May 2025 I assisted @mpopov@wikimedia.org to use that `python3-devel` image for a downstream usage in a Gitlab repo (Slack thread " [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1339636 (https://phabricator.wikimedia.org/T416452) (owner: 10Muehlenhoff) [09:08:54] 06SRE, 06Collaboration-Services, 10GitLab: "apt update" failing in CI - https://phabricator.wikimedia.org/T437836#12316587 (10Mvolz) Dupe of T437069 maybe? [09:09:39] (03CR) 10Jelto: [C:03+2] etherpad: add helm chart for etherpad service [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327571 (https://phabricator.wikimedia.org/T435509) (owner: 10Jelto) [09:11:51] (03Merged) 10jenkins-bot: etherpad: add helm chart for etherpad service [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327571 (https://phabricator.wikimedia.org/T435509) (owner: 10Jelto) [09:15:02] 06SRE, 06Collaboration-Services, 10GitLab: "apt update" failing in CI - https://phabricator.wikimedia.org/T437836#12316631 (10Mvolz) Ticket says it's about bullseye and because it's EOL, but above is bookworm and I have builds failing on trixie, too... ` > [2/7] RUN apt-get update && apt-get install -y "g... [09:16:43] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 14 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335401 (https://phabricator.wikimedia.org/T437006) (owner: 10Tryvix1509) [09:17:51] 06SRE, 06Collaboration-Services, 10Gerrit, 10GitLab: "apt update" failing in CI, on both bullseye and trixie, and both gerrit and gitlab - https://phabricator.wikimedia.org/T437836#12316665 (10Mvolz) [09:18:00] 06SRE, 06Collaboration-Services, 10Gerrit, 10GitLab: "apt update" failing in CI, on both bookworm and trixie, and both gerrit and gitlab - https://phabricator.wikimedia.org/T437836#12316667 (10Mvolz) [09:28:31] (03CR) 10Daniel Kinzler: [C:03+1] wmf-config: Register content/v2-beta REST module as disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339811 (https://phabricator.wikimedia.org/T432798) (owner: 10Aghirelli) [09:31:19] (03CR) 10Kamila Součková: [C:03+1] php: remove stale images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1340948 (owner: 10Giuseppe Lavagetto) [09:33:03] (03CR) 10Kamila Součková: "Duplicate of I6e576367d354fcc66fab351bea4ed4bade8b2993 , I'm not sure why that isn't merged yet." [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1340948 (owner: 10Giuseppe Lavagetto) [09:33:20] (03CR) 10Kamila Součková: [C:03+1] Stop the build of Bullseye PHP base images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1337932 (https://phabricator.wikimedia.org/T417389) (owner: 10Muehlenhoff) [09:40:01] !log drop links tables from db1228 (T437278) [09:40:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:40:05] T437278: Drop unneeded tables from x4 and s4 - https://phabricator.wikimedia.org/T437278 [09:41:46] !log drop links tables from db2172 (T437278) [09:41:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:43:59] (03CR) 10Santiago Faci: [C:03+2] Test Kitchen UI: Deploy v1.5.5 release to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341040 (https://phabricator.wikimedia.org/T429524) (owner: 10Santiago Faci) [09:46:34] (03Merged) 10jenkins-bot: Test Kitchen UI: Deploy v1.5.5 release to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341040 (https://phabricator.wikimedia.org/T429524) (owner: 10Santiago Faci) [09:47:17] (03PS1) 10Ayounsi: Add Arelion Prometheus integration [puppet] - 10https://gerrit.wikimedia.org/r/1341098 (https://phabricator.wikimedia.org/T311005) [09:47:50] (03PS2) 10Ayounsi: Add Arelion Prometheus integration [puppet] - 10https://gerrit.wikimedia.org/r/1341098 (https://phabricator.wikimedia.org/T311005) [09:49:02] (03CR) 10CWilliams: "Should this be accompanied by cumin1003 having transfer disabled whilst it is still an active instance? At a glance, it would seem that we" [puppet] - 10https://gerrit.wikimedia.org/r/1339778 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [09:50:17] (03CR) 10CI reject: [V:04-1] Add Arelion Prometheus integration [puppet] - 10https://gerrit.wikimedia.org/r/1341098 (https://phabricator.wikimedia.org/T311005) (owner: 10Ayounsi) [09:51:15] 06SRE, 06Collaboration-Services, 10Gerrit, 10GitLab: "apt update" failing in CI, on both bookworm and trixie, and both gerrit and gitlab - https://phabricator.wikimedia.org/T437836#12316817 (10hashar) →14Duplicate dup:03T437069 [09:51:16] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review, 06Release-Engineering-Team (Radar): apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12316814 (10hashar) [09:52:19] 06SRE, 06Collaboration-Services, 10GitLab: "apt update" failing in CI, on both bookworm and trixie, and both gerrit and gitlab - https://phabricator.wikimedia.org/T437836#12316821 (10hashar) This is a dupe of T437069 / T437829 (both have more informations and attached patches). It is not related to Gerr... [09:53:07] (03CR) 10Jelto: [C:03+2] "I'll look into this after initial deployment in eqiad has been verified." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327572 (https://phabricator.wikimedia.org/T435509) (owner: 10Jelto) [09:53:12] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply [09:53:20] (03CR) 10CI reject: [V:04-1] helmfile.d: deploy etherpad to aux-k8s clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327572 (https://phabricator.wikimedia.org/T435509) (owner: 10Jelto) [09:53:23] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply [09:54:34] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review, 06Release-Engineering-Team (Radar): apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12316829 (10hashar) There was a related issue happening this morning mentioning expired release files. T... [09:56:12] 06SRE, 06Collaboration-Services, 13Patch-For-Review: setup gerrit2003 with gerrit service (gerrit on bookworm) - https://phabricator.wikimedia.org/T372804#12316840 (10hashar) [09:58:04] 10SRE-swift-storage, 10Ceph, 06Infrastructure-Foundations: Move the majority of the Registry's docker image prefixes to a new s3 bucket - https://phabricator.wikimedia.org/T435499#12316841 (10elukey) 05In progress→03Resolved Going to call this done since I didn't see any issue so far. Please re-open... [09:58:32] (03PS1) 10Brouberol: dse-k8s-eqiad/mediawiki-dumps-legacy: increase max cpu limits [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341100 [09:59:24] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to production (ops group) for dkertesz - https://phabricator.wikimedia.org/T437835#12316846 (10Fabfur) [09:59:41] (03PS10) 10Jelto: helmfile.d: deploy etherpad to aux-k8s clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327572 (https://phabricator.wikimedia.org/T435509) [09:59:46] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to production (ops group) for dkertesz - https://phabricator.wikimedia.org/T437835#12316847 (10Fabfur) Waiting for @ssingh confirmation and we can move on with `ops` group! [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260914T1000) [10:02:07] (03CR) 10Marostegui: [C:03+1] sre.mysql.decommission: optional DC-ops handover [cookbooks] - 10https://gerrit.wikimedia.org/r/1328570 (https://phabricator.wikimedia.org/T435912) (owner: 10Federico Ceratto) [10:02:51] 10SRE-swift-storage, 10Ceph, 06Infrastructure-Foundations, 13Patch-For-Review: Create new S3 backends for the Docker Registry service - https://phabricator.wikimedia.org/T427175#12316858 (10elukey) The Registry has been working using S3/apus as backend for a week and no issue was registered! Metrics look g... [10:04:22] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply [10:04:50] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply [10:05:16] (03PS3) 10Ayounsi: Add Arelion Prometheus integration [puppet] - 10https://gerrit.wikimedia.org/r/1341098 (https://phabricator.wikimedia.org/T311005) [10:06:40] FIRING: SystemdUnitFailed: production-images-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:13:01] (03CR) 10Federico Ceratto: [C:03+2] sre.mysql.decommission: optional DC-ops handover [cookbooks] - 10https://gerrit.wikimedia.org/r/1328570 (https://phabricator.wikimedia.org/T435912) (owner: 10Federico Ceratto) [10:19:05] (03CR) 10Giuseppe Lavagetto: profile::kubernetes::node: Add gvisor labels (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1330187 (https://phabricator.wikimedia.org/T436212) (owner: 10Giuseppe Lavagetto) [10:19:16] (03CR) 10Giuseppe Lavagetto: [C:03+2] profile::kubernetes::node: Add gvisor labels [puppet] - 10https://gerrit.wikimedia.org/r/1330187 (https://phabricator.wikimedia.org/T436212) (owner: 10Giuseppe Lavagetto) [10:22:01] (03CR) 10Btullis: [C:03+1] dse-k8s-eqiad/mediawiki-dumps-legacy: increase max cpu limits [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341100 (owner: 10Brouberol) [10:25:26] (03CR) 10Brouberol: [C:03+2] dse-k8s-eqiad/mediawiki-dumps-legacy: increase max cpu limits [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341100 (owner: 10Brouberol) [10:26:48] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [10:27:38] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [10:37:02] !log marostegui@cumin1004 START - Cookbook sre.mysql.pool pool db1260: After cloning [10:37:12] (03PS1) 10Ladsgroup: Update dbstore DNS record for x4 [dns] - 10https://gerrit.wikimedia.org/r/1341108 (https://phabricator.wikimedia.org/T437839) [10:37:29] !log marostegui@cumin1004 conftool action : set/pooled=no; selector: name=clouddb1025.eqiad.wmnet,service=x4 [10:40:51] (03CR) 10Marostegui: [C:03+1] Update dbstore DNS record for x4 [dns] - 10https://gerrit.wikimedia.org/r/1341108 (https://phabricator.wikimedia.org/T437839) (owner: 10Ladsgroup) [10:42:22] (03CR) 10Ladsgroup: [C:03+2] Update dbstore DNS record for x4 [dns] - 10https://gerrit.wikimedia.org/r/1341108 (https://phabricator.wikimedia.org/T437839) (owner: 10Ladsgroup) [10:42:51] !log ladsgroup@dns1004 START - running authdns-update [10:44:26] (03PS3) 10JavierMonton: stream: pageview-trending-relative [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333854 (https://phabricator.wikimedia.org/T431555) [10:45:02] !log ladsgroup@dns1004 END - running authdns-update [10:48:29] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [10:48:35] (03CR) 10Btullis: [C:03+2] ceph: Make the CephX import guard compare the key [puppet] - 10https://gerrit.wikimedia.org/r/1337876 (https://phabricator.wikimedia.org/T437233) (owner: 10Btullis) [10:50:12] (03PS1) 10JavierMonton: stream: mw-page-html-content-change-enrich [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341109 (https://phabricator.wikimedia.org/T420217) [10:51:23] (03CR) 10Marostegui: "but db1174 is appearing in zarcillo as not pooled, shouldn't that be fixed too? If it is not in dbctl why would it appear there?" [alerts] - 10https://gerrit.wikimedia.org/r/1337624 (https://phabricator.wikimedia.org/T436051) (owner: 10Federico Ceratto) [10:54:22] (03PS1) 10Muehlenhoff: build-bare-slim: Avoid snapshot.debian.org by using --non-debian (WIP) [puppet] - 10https://gerrit.wikimedia.org/r/1341110 (https://phabricator.wikimedia.org/T437829) [10:56:02] (03CR) 10Federico Ceratto: "I updated zarcillo to stop generating metrics for hosts not in dbctl." [alerts] - 10https://gerrit.wikimedia.org/r/1337624 (https://phabricator.wikimedia.org/T436051) (owner: 10Federico Ceratto) [10:59:24] (03CR) 10Marostegui: "When would that take effect? Just checking because zarcillo shows it now." [alerts] - 10https://gerrit.wikimedia.org/r/1337624 (https://phabricator.wikimedia.org/T436051) (owner: 10Federico Ceratto) [11:03:54] (03PS1) 10Marostegui: an-redacteddb1001.yaml: Add x4 [puppet] - 10https://gerrit.wikimedia.org/r/1341115 (https://phabricator.wikimedia.org/T437838) [11:04:44] (03CR) 10Federico Ceratto: "The change is in the generated metric and I see db1174 disappearing https://grafana.wikimedia.org/goto/s4hg76?orgId=default (BTW maybe we " [alerts] - 10https://gerrit.wikimedia.org/r/1337624 (https://phabricator.wikimedia.org/T436051) (owner: 10Federico Ceratto) [11:06:00] (03CR) 10Marostegui: "As I mention, it is on zarcillo where it keeps showing, is that expected?" [alerts] - 10https://gerrit.wikimedia.org/r/1337624 (https://phabricator.wikimedia.org/T436051) (owner: 10Federico Ceratto) [11:10:29] (03CR) 10Muehlenhoff: "That's a good point. I had a look in git history and back in 2025 when DB backups were moved from cumin1002 -> cumin1003 that was also do" [puppet] - 10https://gerrit.wikimedia.org/r/1339778 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [11:15:11] (03CR) 10CWilliams: [C:03+1] "LGTM for now, we can review settings at a later date if needed." [puppet] - 10https://gerrit.wikimedia.org/r/1337601 (https://phabricator.wikimedia.org/T435059) (owner: 10Federico Ceratto) [11:16:13] (03PS11) 10Jelto: helmfile.d: deploy etherpad to aux-k8s clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327572 (https://phabricator.wikimedia.org/T435509) [11:17:06] (03CR) 10CWilliams: "Sounds good to me, thanks." [puppet] - 10https://gerrit.wikimedia.org/r/1339778 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [11:22:08] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1260: After cloning [11:22:10] (03PS4) 10Muehlenhoff: Move DB backups from cumin1003 to cumin1004 [puppet] - 10https://gerrit.wikimedia.org/r/1339778 (https://phabricator.wikimedia.org/T427897) [11:24:18] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1339778 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [11:24:50] 06SRE, 10corto, 10Incident Tooling: Increase trusted volunteer's visibility into production incidents - https://phabricator.wikimedia.org/T426137#12317189 (10Bugreporter) >>! In T426137#12277753, @SomeRandomDeveloper wrote: >>>! In T426137#12276917, @Novem_Linguae wrote: >> @SomeRandomDeveloper. Sounds like... [11:26:05] (03PS1) 10Muehlenhoff: profile::dbbackups::transfer: Default db transfers to false [puppet] - 10https://gerrit.wikimedia.org/r/1341121 (https://phabricator.wikimedia.org/T427897) [11:26:27] (03PS1) 10Btullis: ceph: Re-enable manage_keydata for the DPE Ceph cluster in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1341122 (https://phabricator.wikimedia.org/T437447) [11:28:14] (03PS2) 10Btullis: ceph: Re-enable manage_keydata for the DPE Ceph cluster in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1341122 (https://phabricator.wikimedia.org/T437447) [11:28:26] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1341122 (https://phabricator.wikimedia.org/T437447) (owner: 10Btullis) [11:30:06] (03PS5) 10Muehlenhoff: Move DB backups from cumin1003 to cumin1004 [puppet] - 10https://gerrit.wikimedia.org/r/1339778 (https://phabricator.wikimedia.org/T427897) [11:32:52] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1339778 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [11:35:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 13.26% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:35:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:40:38] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for db1245.mgmt:22 - https://phabricator.wikimedia.org/T437778#12317249 (10Jclark-ctr) Rebooted idrac reachable again [11:40:40] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for db1245.mgmt:22 - https://phabricator.wikimedia.org/T437778#12317250 (10Jclark-ctr) 05Open→03Resolved [11:42:00] (03CR) 10Marostegui: [C:03+2] an-redacteddb1001.yaml: Add x4 [puppet] - 10https://gerrit.wikimedia.org/r/1341115 (https://phabricator.wikimedia.org/T437838) (owner: 10Marostegui) [11:44:41] (03PS1) 10Dpogorzelski: dns: add liftwing-studio records [dns] - 10https://gerrit.wikimedia.org/r/1341128 (https://phabricator.wikimedia.org/T437706) [11:49:37] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:53:24] (03PS2) 10Slyngshede: P:idp enable MFA for production [puppet] - 10https://gerrit.wikimedia.org/r/1326833 (https://phabricator.wikimedia.org/T277841) [11:55:29] (03PS1) 10Dpogorzelski: liftwing-studio: add to service catalog [puppet] - 10https://gerrit.wikimedia.org/r/1341130 (https://phabricator.wikimedia.org/T437706) [11:56:00] (03PS2) 10Muehlenhoff: profile::dbbackups::transfer: Default db transfers to false [puppet] - 10https://gerrit.wikimedia.org/r/1341121 (https://phabricator.wikimedia.org/T427897) [11:56:25] (03PS1) 10Jelto: service::catalog: Set ipip for recommendation-api,sessionstore,tegola,termbox [puppet] - 10https://gerrit.wikimedia.org/r/1341131 (https://phabricator.wikimedia.org/T420436) [11:56:27] (03PS1) 10Jelto: service::catalog: Set ipip for recommendation-api,sessionstore,tegola,termbox [puppet] - 10https://gerrit.wikimedia.org/r/1341132 (https://phabricator.wikimedia.org/T420436) [11:58:39] (03PS1) 10Slyngshede: P:idp enabled optional MFA for Puppetboard [puppet] - 10https://gerrit.wikimedia.org/r/1341133 (https://phabricator.wikimedia.org/T437879) [11:59:31] (03CR) 10Joal: [C:03+1] stream: mw-page-html-content-change-enrich [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341109 (https://phabricator.wikimedia.org/T420217) (owner: 10JavierMonton) [11:59:38] (03CR) 10Joal: [C:03+1] stream: pageview-trending-relative [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333854 (https://phabricator.wikimedia.org/T431555) (owner: 10JavierMonton) [12:02:00] PROBLEM - mysqld processes on an-redacteddb1001 is CRITICAL: PROCS CRITICAL: 8 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [12:02:19] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T437170#12317332 (10Jclark-ctr) 05Open→03Resolved Returning drive to dell will request replacement again if fails again [12:03:37] (03CR) 10Muehlenhoff: [C:03+1] "LGTM (once group approval is in)" [puppet] - 10https://gerrit.wikimedia.org/r/1339866 (https://phabricator.wikimedia.org/T437652) (owner: 10Ssingh) [12:04:01] (03CR) 10Arnaudb: [C:03+1] "that looks good to me, +1!" [puppet] - 10https://gerrit.wikimedia.org/r/1331623 (https://phabricator.wikimedia.org/T435509) (owner: 10Jelto) [12:04:41] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1339879 (https://phabricator.wikimedia.org/T437653) (owner: 10Ssingh) [12:04:45] (03CR) 10Jelto: "recheck" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327572 (https://phabricator.wikimedia.org/T435509) (owner: 10Jelto) [12:05:32] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1341121 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [12:08:48] (03CR) 10Jelto: [C:03+2] helmfile.d: deploy etherpad to aux-k8s clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327572 (https://phabricator.wikimedia.org/T435509) (owner: 10Jelto) [12:11:08] PROBLEM - MariaDB Replica IO: x4 on an-redacteddb1001 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [12:11:36] (03Merged) 10jenkins-bot: helmfile.d: deploy etherpad to aux-k8s clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327572 (https://phabricator.wikimedia.org/T435509) (owner: 10Jelto) [12:12:06] (03PS1) 10Ayounsi: Add mock secrets for Arelion API [labs/private] - 10https://gerrit.wikimedia.org/r/1341156 (https://phabricator.wikimedia.org/T311005) [12:12:57] (03PS4) 10Ayounsi: Add Arelion Prometheus integration [puppet] - 10https://gerrit.wikimedia.org/r/1341098 (https://phabricator.wikimedia.org/T311005) [12:14:41] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply [12:14:47] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply [12:14:56] (03PS3) 10Sbisson: ArticleGuidance: Remove the experiment configuration keys [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1334946 (https://phabricator.wikimedia.org/T434487) [12:15:09] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware), 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12317356 (10fgiunchedi) FWIW I agree with the proposal! Is there any reason why we went with hw raid1 for the ssd/nvme on these hosts... [12:20:07] (03PS1) 10Brouberol: kerberized-apps: add analytics-test to the helmfile list [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341159 [12:20:10] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [labs/private] - 10https://gerrit.wikimedia.org/r/1341156 (https://phabricator.wikimedia.org/T311005) (owner: 10Ayounsi) [12:20:47] (03CR) 10Ayounsi: [V:03+2 C:03+2] Add mock secrets for Arelion API [labs/private] - 10https://gerrit.wikimedia.org/r/1341156 (https://phabricator.wikimedia.org/T311005) (owner: 10Ayounsi) [12:21:08] PROBLEM - MariaDB Replica SQL: x4 on an-redacteddb1001 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [12:21:34] (03CR) 10Ayounsi: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1341098 (https://phabricator.wikimedia.org/T311005) (owner: 10Ayounsi) [12:21:51] (03PS1) 10Urbanecm: [Growth] Remove unused config variables [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341161 (https://phabricator.wikimedia.org/T392944) [12:21:53] (03CR) 10Dbrant: [C:03+2] wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1339781 (owner: 10PipelineBot) [12:22:48] (03CR) 10Brouberol: [C:03+1] ceph: Re-enable manage_keydata for the DPE Ceph cluster in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1341122 (https://phabricator.wikimedia.org/T437447) (owner: 10Btullis) [12:23:08] PROBLEM - MariaDB Replica Lag: x4 on an-redacteddb1001 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [12:23:14] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 14 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1334946 (https://phabricator.wikimedia.org/T434487) (owner: 10Sbisson) [12:23:43] (03CR) 10Btullis: [C:03+1] kerberized-apps: add analytics-test to the helmfile list [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341159 (owner: 10Brouberol) [12:24:20] (03CR) 10Urbanecm: [C:04-2] "To be shipped next week, when the removing patch is in prod." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341161 (https://phabricator.wikimedia.org/T392944) (owner: 10Urbanecm) [12:24:38] (03Merged) 10jenkins-bot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1339781 (owner: 10PipelineBot) [12:24:39] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.7 point update - https://phabricator.wikimedia.org/T437715#12317389 (10MoritzMuehlenhoff) [12:24:50] (03CR) 10Giuseppe Lavagetto: kubernetes::staging: make containerd support gVisor runtime (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1338189 (owner: 10Giuseppe Lavagetto) [12:25:38] (03PS1) 10Dpogorzelski: services_proxy: send SNI on the inference listener [puppet] - 10https://gerrit.wikimedia.org/r/1341163 (https://phabricator.wikimedia.org/T437706) [12:25:41] (03CR) 10Giuseppe Lavagetto: [C:03+2] admin: add support for gVisor RuntimeClass handlers [deployment-charts] - 10https://gerrit.wikimedia.org/r/1330344 (https://phabricator.wikimedia.org/T436212) (owner: 10Giuseppe Lavagetto) [12:26:08] PROBLEM - MariaDB read only wikireplica-x4 on an-redacteddb1001 is CRITICAL: Could not connect to localhost:3364 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [12:26:40] (03Abandoned) 10Dbrant: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334323 (owner: 10PipelineBot) [12:27:00] (03PS2) 10Dpogorzelski: services_proxy: send SNI on the inference listener [puppet] - 10https://gerrit.wikimedia.org/r/1341163 (https://phabricator.wikimedia.org/T437706) [12:29:24] (03CR) 10Brouberol: [C:03+2] kerberized-apps: add analytics-test to the helmfile list [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341159 (owner: 10Brouberol) [12:29:29] (03CR) 10Ladsgroup: [C:03+2] thumbor: Actually enable asyncio [deployment-charts] - 10https://gerrit.wikimedia.org/r/1339015 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup) [12:30:29] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply [12:31:08] PROBLEM - MariaDB read only x4 on an-redacteddb1001 is CRITICAL: Could not connect to localhost:3364 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [12:31:10] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Rebalance cloudvirts out of F4 and into D5 - https://phabricator.wikimedia.org/T435921#12317406 (10fgiunchedi) 05In progress→03Resolved Hosts are back in service [12:31:22] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Move cloudvirt1075 from F4 to E4 - https://phabricator.wikimedia.org/T435923#12317411 (10fgiunchedi) 05Open→03Resolved Host is back in service [12:34:16] <_joe_> !log adding gvisor labels to all wikikube clusters nodes [12:34:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:34:52] 06SRE, 10corto, 10Incident Tooling: Increase trusted volunteer's visibility into production incidents - https://phabricator.wikimedia.org/T426137#12317428 (10Johannnes89) >>! In T426137#12317189, @Bugreporter wrote: >>>! In T426137#12277753, @SomeRandomDeveloper wrote: >>>>! In T426137#12276917, @Novem_Lingu... [12:35:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.73% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:35:30] (03Merged) 10jenkins-bot: admin: add support for gVisor RuntimeClass handlers [deployment-charts] - 10https://gerrit.wikimedia.org/r/1330344 (https://phabricator.wikimedia.org/T436212) (owner: 10Giuseppe Lavagetto) [12:35:55] (03PS1) 10Brouberol: helmfile_pending_changes: only report from a single site [alerts] - 10https://gerrit.wikimedia.org/r/1341168 [12:36:48] !log marostegui@cumin1004 conftool action : set/pooled=yes; selector: name=clouddb1025.eqiad.wmnet,service=x4 [12:37:00] RECOVERY - mysqld processes on an-redacteddb1001 is OK: PROCS OK: 9 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [12:37:08] RECOVERY - MariaDB read only x4 on an-redacteddb1001 is OK: Version 10.11.16-MariaDB, Uptime 13s, read_only: True, event_scheduler: False, 17.92 QPS, connection latency: 0.038794s, query latency: 0.000621s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [12:37:08] RECOVERY - MariaDB read only wikireplica-x4 on an-redacteddb1001 is OK: Version 10.11.16-MariaDB, Uptime 13s, read_only: True, event_scheduler: False, 22.39 QPS, connection latency: 0.046001s, query latency: 0.000664s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [12:37:29] !log dbrant@deploy1003 helmfile [staging] START helmfile.d/services/wikifeeds: apply [12:37:49] !log dbrant@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifeeds: apply [12:38:08] RECOVERY - MariaDB Replica Lag: x4 on an-redacteddb1001 is OK: OK slave_sql_lag not a slave https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [12:38:08] RECOVERY - MariaDB Replica IO: x4 on an-redacteddb1001 is OK: OK slave_io_state not a slave https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [12:38:08] RECOVERY - MariaDB Replica SQL: x4 on an-redacteddb1001 is OK: OK slave_sql_state not a slave https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [12:38:38] (03CR) 10Aklapper: [V:03+2 C:03+2] "Thanks. Applies cleanly locally" [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1338245 (owner: 10Pppery) [12:39:17] !log dbrant@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifeeds: apply [12:39:22] (03CR) 10Atsuko: [C:03+1] helmfile_pending_changes: only report from a single site [alerts] - 10https://gerrit.wikimedia.org/r/1341168 (owner: 10Brouberol) [12:39:44] !log dbrant@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply [12:39:48] (03CR) 10Brouberol: [C:03+2] helmfile_pending_changes: only report from a single site [alerts] - 10https://gerrit.wikimedia.org/r/1341168 (owner: 10Brouberol) [12:39:55] !log dbrant@deploy1003 helmfile [codfw] START helmfile.d/services/wikifeeds: apply [12:40:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.14% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:40:19] !log oblivian@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'. [12:40:23] !log dbrant@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply [12:40:34] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply [12:40:37] (03PS1) 10Marostegui: check_private_data_report: Add x4 on an-redacteddb1001 [puppet] - 10https://gerrit.wikimedia.org/r/1341170 (https://phabricator.wikimedia.org/T437838) [12:41:27] (03CR) 10Marostegui: [C:03+2] check_private_data_report: Add x4 on an-redacteddb1001 [puppet] - 10https://gerrit.wikimedia.org/r/1341170 (https://phabricator.wikimedia.org/T437838) (owner: 10Marostegui) [12:41:45] (03Merged) 10jenkins-bot: thumbor: Actually enable asyncio [deployment-charts] - 10https://gerrit.wikimedia.org/r/1339015 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup) [12:42:27] !log oblivian@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [12:44:16] !log ladsgroup@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [12:44:34] !log ladsgroup@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [12:45:11] (03PS1) 10Dpogorzelski: liftwing-studio: chart deployment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341174 (https://phabricator.wikimedia.org/T437706) [12:45:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.9% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:47:54] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply [12:48:06] !log ladsgroup@deploy1003 helmfile [codfw] START helmfile.d/services/thumbor: apply [12:48:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.24% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:48:51] (03PS6) 10Giuseppe Lavagetto: kubernetes::staging: make containerd support gVisor runtime [puppet] - 10https://gerrit.wikimedia.org/r/1338189 [12:49:35] !log ladsgroup@deploy1003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [12:49:39] (03CR) 10Btullis: [C:03+2] ceph: Re-enable manage_keydata for the DPE Ceph cluster in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1341122 (https://phabricator.wikimedia.org/T437447) (owner: 10Btullis) [12:49:51] 06SRE, 10corto, 10Incident Tooling: Increase trusted volunteer's visibility into production incidents - https://phabricator.wikimedia.org/T426137#12317477 (10Bugreporter) When I see stewards got security task access years ago, I was wondering whether we should require all elected stewards to sign Volunteer N... [12:51:12] (03PS1) 10Brouberol: cloudnative-pg/CloudnativePGWalArchivingFailing: don't alert for clusters which never encountered a failure [alerts] - 10https://gerrit.wikimedia.org/r/1341180 [12:51:15] (03CR) 10JavierMonton: [C:03+2] stream: mw-page-html-content-change-enrich [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341109 (https://phabricator.wikimedia.org/T420217) (owner: 10JavierMonton) [12:52:21] !log ladsgroup@deploy1003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [12:52:52] (03CR) 10Giuseppe Lavagetto: [C:03+2] kubernetes::staging: make containerd support gVisor runtime [puppet] - 10https://gerrit.wikimedia.org/r/1338189 (owner: 10Giuseppe Lavagetto) [12:52:59] (03CR) 10Kamila Součková: [C:04-1] "The duplicate was merged, so this is not needed anymore." [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1340948 (owner: 10Giuseppe Lavagetto) [12:53:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.58% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:54:09] (03Merged) 10jenkins-bot: stream: mw-page-html-content-change-enrich [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341109 (https://phabricator.wikimedia.org/T420217) (owner: 10JavierMonton) [12:54:21] !log ladsgroup@deploy1003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [12:54:41] (03CR) 10Btullis: [C:03+1] cloudnative-pg/CloudnativePGWalArchivingFailing: don't alert for clusters which never encountered a failure [alerts] - 10https://gerrit.wikimedia.org/r/1341180 (owner: 10Brouberol) [12:54:48] (03PS1) 10JMeybohm: k8s: Add a script to sync kubelet labels with the API server [puppet] - 10https://gerrit.wikimedia.org/r/1341181 [12:55:18] (03CR) 10Brouberol: [C:03+2] cloudnative-pg/CloudnativePGWalArchivingFailing: don't alert for clusters which never encountered a failure [alerts] - 10https://gerrit.wikimedia.org/r/1341180 (owner: 10Brouberol) [12:55:23] (03CR) 10CI reject: [V:04-1] k8s: Add a script to sync kubelet labels with the API server [puppet] - 10https://gerrit.wikimedia.org/r/1341181 (owner: 10JMeybohm) [12:56:18] (03CR) 10Ladsgroup: "recheck" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1339064 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup-claude) [12:56:37] (03CR) 10Ladsgroup: [WIP] Set JPGs to be thumbnailed via VIPS if md5 starts with a (031 comment) [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1339064 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup-claude) [12:56:58] (03CR) 10CI reject: [V:04-1] [WIP] Set JPGs to be thumbnailed via VIPS if md5 starts with a [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1339064 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup-claude) [12:58:00] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply [12:58:55] !log update Trixie installer image to 13.7 T437715 [12:58:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:58:58] T437715: Integrate Trixie 13.7 point update - https://phabricator.wikimedia.org/T437715 [12:59:30] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.7 point update - https://phabricator.wikimedia.org/T437715#12317572 (10MoritzMuehlenhoff) [12:59:50] (03CR) 10Andrew Bogott: "I'm not sure this is useful; I couldn't see any performance improvements from increasing ram allocation" [puppet] - 10https://gerrit.wikimedia.org/r/1338257 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [13:00:04] Lucas_WMDE, urbanecm, and TheresNoTime: Your horoscope predicts another UTC afternoon backport window deploy. May Zuul be (nice) with you. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260914T1300). [13:00:04] Hide_on_rosie, Daimona, bunnypranav, tgr, and stephanebisson: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:09] o/ [13:00:12] I’‘m in a meeting, sorry [13:00:21] (03PS2) 10JMeybohm: k8s: Add a script to sync kubelet labels with the API server [puppet] - 10https://gerrit.wikimedia.org/r/1341181 [13:00:21] o/ [13:00:56] (03PS1) 10Kevin Bazira: ml-services: update tts-section-generator image and ruleset [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341182 (https://phabricator.wikimedia.org/T436758) [13:00:57] (03CR) 10CI reject: [V:04-1] k8s: Add a script to sync kubelet labels with the API server [puppet] - 10https://gerrit.wikimedia.org/r/1341181 (owner: 10JMeybohm) [13:02:28] (03CR) 10Ladsgroup: [C:03+2] Enable $wgCampaignEventsEnableWorklistEventDiscoveryTracking in beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340213 (https://phabricator.wikimedia.org/T434512) (owner: 10Daimona Eaytoy) [13:02:38] that's beta cluster, I'll just merge it [13:03:00] but I'm not sure if we can deploy anything, the base image can't be built. https://integration.wikimedia.org/ci/job/thumbor-plugins-pipeline-test/481/console [13:03:30] (03Merged) 10jenkins-bot: Enable $wgCampaignEventsEnableWorklistEventDiscoveryTracking in beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340213 (https://phabricator.wikimedia.org/T434512) (owner: 10Daimona Eaytoy) [13:03:41] (03CR) 10Muehlenhoff: "The current testkitchen images are based on Trixie (repos/data-engineering/test-kitchen:v1.5.3-prod) and use Python 3.13" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1339636 (https://phabricator.wikimedia.org/T416452) (owner: 10Muehlenhoff) [13:03:50] (03CR) 10Muehlenhoff: [C:03+2] Remove puppet/config-master records pointing to puppetmaster1001 [dns] - 10https://gerrit.wikimedia.org/r/1337819 (https://phabricator.wikimedia.org/T416606) (owner: 10Muehlenhoff) [13:04:08] (03CR) 10Giuseppe Lavagetto: [C:03+2] "Done" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333210 (https://phabricator.wikimedia.org/T436655) (owner: 10Giuseppe Lavagetto) [13:04:15] !log jmm@dns1004 START - running authdns-update [13:05:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:06:23] (03PS3) 10JMeybohm: k8s: Add a script to sync kubelet labels with the API server [puppet] - 10https://gerrit.wikimedia.org/r/1341181 [13:06:28] !log jmm@dns1004 END - running authdns-update [13:07:09] How about other wmf-config patches [13:07:26] (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1341181 (owner: 10JMeybohm) [13:08:25] (03CR) 10Atsuko: [C:03+2] airflow: pluggable authmanager for K8s tokens [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338217 (https://phabricator.wikimedia.org/T437236) (owner: 10Atsuko) [13:08:30] (03CR) 10Atsuko: [C:03+2] airflow: service account projected volume [deployment-charts] - 10https://gerrit.wikimedia.org/r/1339714 (owner: 10Atsuko) [13:08:46] Hi [13:08:53] Sorry I'm a bit late [13:08:55] (03PS1) 10Elukey: admin_ng: move cfssl's issuer endpoint to PKI hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341183 (https://phabricator.wikimedia.org/T436809) [13:09:15] !log sudo cumin "A:cp-text" "disable-puppet 'merging CR 1338134'" [13:09:15] Is there any deployer who can help me deploy mine? [13:09:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:09:39] (03CR) 10Arnaudb: "adding @ffurnari@wikimedia.org to the reviewer list per our chat with @ssingh@wikimedia.org" [puppet] - 10https://gerrit.wikimedia.org/r/1338946 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [13:11:41] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware), 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12317656 (10Andrew) >>! In T419892#12317356, @fgiunchedi wrote: > FWIW I agree with the proposal! Is there any reason why we went wit... [13:12:25] (Very sorry, not available to deploy at the moment) [13:13:47] (03CR) 10Ssingh: [C:03+2] cache: pipe websocket upgrades for the gitlab hostnames [puppet] - 10https://gerrit.wikimedia.org/r/1338134 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [13:14:09] !log installing Bird security updates [13:14:10] bunnypranav: I'm not sure if we can deploy anything atm [13:14:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:14:28] (03CR) 10Brouberol: liftwing-studio: chart deployment (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341174 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [13:14:34] (03CR) 10Brouberol: [C:03+1] liftwing-studio: chart for Open WebUI and LiteLLM [deployment-charts] - 10https://gerrit.wikimedia.org/r/1339756 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [13:15:36] (03Merged) 10jenkins-bot: validating-admission-policies: add gVisor enforcement policy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333210 (https://phabricator.wikimedia.org/T436655) (owner: 10Giuseppe Lavagetto) [13:17:07] (03CR) 10Ozge: [C:03+1] ml-services: update tts-section-generator image and ruleset [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341182 (https://phabricator.wikimedia.org/T436758) (owner: 10Kevin Bazira) [13:17:33] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host cumin1003.eqiad.wmnet [13:17:41] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to production (ops group) for dkertesz - https://phabricator.wikimedia.org/T437835#12317688 (10ssingh) Sounds good. I will get Mark's confirmation on the CR. [13:18:50] PROBLEM - Druid historical on an-druid1006 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [13:18:52] (03CR) 10Brouberol: [C:03+1] liftwing-studio: add to service catalog [puppet] - 10https://gerrit.wikimedia.org/r/1341130 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [13:19:09] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply [13:19:21] (03CR) 10Brouberol: [C:03+1] services_proxy: send SNI on the inference listener [puppet] - 10https://gerrit.wikimedia.org/r/1341163 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [13:19:43] (03CR) 10Brouberol: [C:03+1] dns: add liftwing-studio records [dns] - 10https://gerrit.wikimedia.org/r/1341128 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [13:20:02] (03PS1) 10Arendpieter: Logout: End the OpenID Connect session at the provider [software/bitu] - 10https://gerrit.wikimedia.org/r/1341185 (https://phabricator.wikimedia.org/T392350) [13:20:16] (03Merged) 10jenkins-bot: airflow: pluggable authmanager for K8s tokens [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338217 (https://phabricator.wikimedia.org/T437236) (owner: 10Atsuko) [13:20:48] (03Merged) 10jenkins-bot: airflow: service account projected volume [deployment-charts] - 10https://gerrit.wikimedia.org/r/1339714 (owner: 10Atsuko) [13:20:54] !log sudo cumin -b11 "A:cp-text" "run-puppet-agent --enable 'merging CR 1338134'"T425441 [13:20:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:20:58] !log sudo cumin -b11 "A:cp-text" "run-puppet-agent --enable 'merging CR 1338134'" T425441 [13:20:58] T425441: gitlab behind CDN - https://phabricator.wikimedia.org/T425441 [13:21:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:22:37] (03CR) 10CI reject: [V:04-1] Logout: End the OpenID Connect session at the provider [software/bitu] - 10https://gerrit.wikimedia.org/r/1341185 (https://phabricator.wikimedia.org/T392350) (owner: 10Arendpieter) [13:22:41] (03PS2) 10Arnaudb: gitlab: log the real client address behind the CDN [puppet] - 10https://gerrit.wikimedia.org/r/1341184 (https://phabricator.wikimedia.org/T425441) [13:23:13] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cumin1003.eqiad.wmnet [13:24:37] (03PS2) 10Elukey: admin_ng: move cfssl's issuer endpoint to PKI hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341183 (https://phabricator.wikimedia.org/T436809) [13:25:06] (03CR) 10Ssingh: "Looks good "approved". Please wait for Mark's approval." [puppet] - 10https://gerrit.wikimedia.org/r/1341006 (https://phabricator.wikimedia.org/T437835) (owner: 10Daniel Kertesz) [13:25:45] (03PS1) 10Filippo Giunchedi: wmcs: export prometheus metrics from wmcs-backup [puppet] - 10https://gerrit.wikimedia.org/r/1341187 (https://phabricator.wikimedia.org/T428893) [13:27:06] (03PS1) 10Arendpieter: apereo_cas: Allow CAS to redirect clients after logout [puppet] - 10https://gerrit.wikimedia.org/r/1341188 (https://phabricator.wikimedia.org/T392350) [13:27:09] (03PS1) 10Arendpieter: apereo_cas: Allow services to declare a logout endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1341189 (https://phabricator.wikimedia.org/T233941) [13:27:59] (03CR) 10CI reject: [V:04-1] wmcs: export prometheus metrics from wmcs-backup [puppet] - 10https://gerrit.wikimedia.org/r/1341187 (https://phabricator.wikimedia.org/T428893) (owner: 10Filippo Giunchedi) [13:29:10] (03CR) 10Kevin Bazira: [C:03+2] ml-services: update tts-section-generator image and ruleset [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341182 (https://phabricator.wikimedia.org/T436758) (owner: 10Kevin Bazira) [13:29:15] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply [13:29:19] (03CR) 10Elukey: "Hey folks! I am planning to move pki.discovery.wment to LVS, and this is an intermediate step to allow more invasive changes without creat" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341183 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [13:31:50] (03Merged) 10jenkins-bot: ml-services: update tts-section-generator image and ruleset [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341182 (https://phabricator.wikimedia.org/T436758) (owner: 10Kevin Bazira) [13:32:04] !log oblivian@deploy1003 helmfile [eqiad] START helmfile.d/admin 'apply'. [13:32:08] !log oblivian@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'apply'. [13:32:13] (03PS2) 10Filippo Giunchedi: wmcs: export prometheus metrics from wmcs-backup [puppet] - 10https://gerrit.wikimedia.org/r/1341187 (https://phabricator.wikimedia.org/T428893) [13:32:17] !log oblivian@deploy1003 helmfile [codfw] START helmfile.d/admin 'apply'. [13:32:28] !log oblivian@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'apply'. [13:32:37] !log oblivian@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [13:32:44] !log oblivian@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [13:32:54] !log oblivian@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'. [13:33:07] !log oblivian@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [13:33:10] Amir1: Is it ok to deploy now? [13:33:15] !log oblivian@deploy1003 helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [13:33:22] !log oblivian@deploy1003 helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [13:33:31] !log oblivian@deploy1003 helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [13:33:46] !log oblivian@deploy1003 helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [13:33:55] !log oblivian@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [13:34:09] !log oblivian@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [13:34:18] !log oblivian@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [13:34:28] !log oblivian@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [13:34:35] !log oblivian@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [13:34:49] !log oblivian@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [13:34:53] !log jayme@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [13:34:57] !log oblivian@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. [13:35:04] !log oblivian@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. [13:35:11] !log oblivian@deploy1003 helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. [13:35:14] !log jayme@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [13:35:25] !log oblivian@deploy1003 helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. [13:35:49] <_joe_> jouncebot: now [13:35:49] For the next 0 hour(s) and 24 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260914T1300) [13:36:28] !log jayme@deploy1003 helmfile [codfw] START helmfile.d/admin 'apply'. [13:36:52] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply [13:37:00] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply [13:37:57] !log jayme@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'apply'. [13:38:13] !log jayme@deploy1003 helmfile [eqiad] START helmfile.d/admin 'apply'. [13:38:25] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply [13:38:27] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply [13:38:33] !log jayme@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'apply'. [13:38:50] RECOVERY - Druid historical on an-druid1006 is OK: PROCS OK: 1 process with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [13:38:58] Let's wait please [13:39:03] !log kevinbazira@deploy1003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [13:39:20] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply [13:39:21] Amir: why do we need thumbor for a MediaWiki deployment? from the discussion on _security it seems like this problem was solved for scap [13:39:22] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply [13:40:01] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply [13:40:06] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply [13:40:37] !log eevans@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sessionstore1004.eqiad.wmnet with reason: Firmware upgrades — T437516 [13:40:44] T437516: sessionstore: Debian Bookworm upgrades - https://phabricator.wikimedia.org/T437516 [13:40:59] !log eevans@cumin1004 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts sessionstore1004.eqiad.wmnet [13:41:34] (03CR) 10Mark Bergsma: [C:03+1] admin: move dkertesz from ops-limited to ops [puppet] - 10https://gerrit.wikimedia.org/r/1341006 (https://phabricator.wikimedia.org/T437835) (owner: 10Daniel Kertesz) [13:41:51] 10SRE-SLO, 06Abstract Wikipedia team (27Q1 (Jul–Sep)), 07OKR-Work: new SLI (1 of 2): server-side metrics on Abstract Wikipedia preview - https://phabricator.wikimedia.org/T434231#12317839 (10DSantamaria) a:03ecarg [13:42:47] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply [13:43:22] tgr_: if it's solved for scap, go for it [13:43:32] I was using thumbor as example [13:43:46] !log kevinbazira@deploy1003 helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . [13:44:31] the task is T437824 btw [13:44:32] T437824: wikifeeds-pipeline-test fails due to expired Debian snapshot repositories - https://phabricator.wikimedia.org/T437824 [13:44:57] !log kevinbazira@deploy1003 helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [13:46:45] I don't have much time, I can do one deploy [13:47:07] (03CR) 10Filippo Giunchedi: "Sample output from cloudbackup2003: https://phabricator.wikimedia.org/P96450" [puppet] - 10https://gerrit.wikimedia.org/r/1341187 (https://phabricator.wikimedia.org/T428893) (owner: 10Filippo Giunchedi) [13:47:22] the CSP one will need a bunch of testing, the other ones look like they could be deployed in one go [13:47:34] Hide_on_rosie: bunnypranav: stephanebisson: ? [13:47:52] ready. [13:47:55] is scap actually fixed? [13:47:56] My patch is a no-op config cleanup. I have confirm with code search that nothing is reading those variables anymore. [13:47:56] WikimediaDebug installed [13:48:12] the patch on T437829 is still open [13:48:13] T437829: scap backport failed due to expired debian snapshot - https://phabricator.wikimedia.org/T437829 [13:48:49] (03CR) 10Fabfur: [C:03+2] admin: move dkertesz from ops-limited to ops [puppet] - 10https://gerrit.wikimedia.org/r/1341006 (https://phabricator.wikimedia.org/T437835) (owner: 10Daniel Kertesz) [13:51:03] taavi: SAL says the scap at 7AM today was successful [13:51:23] IIUIC from mediaiwki_security, there is some limited workaround in place [13:51:26] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to production (ops group) for dkertesz - https://phabricator.wikimedia.org/T437835#12317882 (10dkertesz) 05Open→03Resolved a:03Fabfur CR approved by Mark and merged by Fabrizio [13:51:30] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to production (ops group) for dkertesz - https://phabricator.wikimedia.org/T437835#12317885 (10Fabfur) a:05Fabfur→03None [13:52:20] my internet is acting up though so I'll leave it to someone else though, sorry [13:52:52] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply [13:54:19] I’m around now if needed… [13:54:20] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply [13:54:37] eevans@cumin1004 upgrade-firmware (PID 2105023) is awaiting input [13:54:48] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 14 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339811 (https://phabricator.wikimedia.org/T432798) (owner: 10Aghirelli) [13:56:25] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply [13:58:16] eevans@cumin1004 upgrade-firmware (PID 2105023) is awaiting input [13:59:05] Lucas_WMDE: Hello. [13:59:33] I'll deploy my own patch at 18 UTC, I can deploy others if the owners can be around that time (gtg to a meeting now) [14:00:14] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply [14:00:23] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply [14:00:28] sorry that's 18 CEST / 16 UTC [14:00:54] !log eevans@cumin1004 START - Cookbook sre.hosts.reboot-single for host sessionstore1004.eqiad.wmnet [14:01:25] RESOLVED: SystemdUnitFailed: production-images-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:04:07] FIRING: ProbeDown: Service sessionstore1004-a:9042 has failed probes (tcp_cassandra_a_cql_ip4) - https://wikitech.wikimedia.org/wiki/TLS/Runbook#sessionstore1004-a:9042 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:05:17] !log kick off a rebuild of base images on build2004 [14:05:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:05:32] !log kick off a rebuild of base images on build2004 [14:05:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:09:07] FIRING: [2x] ProbeDown: Service sessionstore1004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:13:00] !log eevans@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore1004.eqiad.wmnet [14:13:01] !log eevans@cumin1004 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts sessionstore1004.eqiad.wmnet [14:13:34] !log eevans@cumin1004 START - Cookbook sre.hosts.provision for host sessionstore1004.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:13:49] sorry, and I promptly got sidetracked 😅 [14:13:51] jouncebot: now [14:13:51] No deployments scheduled for the next 0 hour(s) and 16 minute(s) [14:13:57] hm, test kitchen soon [14:14:07] RESOLVED: [2x] ProbeDown: Service sessionstore1004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:14:13] !log eevans@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sessionstore1004.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:15:40] !log eevans@cumin1004 START - Cookbook sre.hosts.reimage for host sessionstore1004.eqiad.wmnet with OS bookworm [14:16:56] (03CR) 10Lucas Werkmeister (WMDE): [C:04-1] afwiki: Create Draft and Draft talk namespaces (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338902 (https://phabricator.wikimedia.org/T437576) (owner: 10Tryvix1509) [14:20:10] FIRING: [2x] ProbeDown: Service sessionstore1004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:20:24] (03CR) 10Lucas Werkmeister (WMDE): "Not done – if I’m not mistaken, in PS2..3 you accidentally undid the change in the viwiki block (so that this block now has the “older” fo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335401 (https://phabricator.wikimedia.org/T437006) (owner: 10Tryvix1509) [14:20:41] Hide_on_rosie: ^ fyi [14:21:29] Seen it, working [14:21:33] ok [14:21:47] maybe we can try deploying one of them if the test kitchen window isn’t needed [14:21:53] (and see if deployments are still broken or not) [14:24:46] (03PS4) 10Tryvix1509: Adjust extendedconfirmed calculation to first edit on viwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335401 (https://phabricator.wikimedia.org/T437006) [14:25:10] FIRING: [2x] ProbeDown: Service sessionstore1004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:25:25] RESOLVED: [2x] ProbeDown: Service sessionstore1004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:25:56] (03CR) 10Tryvix1509: "Done" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335401 (https://phabricator.wikimedia.org/T437006) (owner: 10Tryvix1509) [14:28:06] (03PS2) 10Tryvix1509: afwiki: Create Draft and Draft talk namespaces [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338902 (https://phabricator.wikimedia.org/T437576) [14:28:53] (03PS3) 10Tryvix1509: afwiki: Create Draft and Draft talk namespaces [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338902 (https://phabricator.wikimedia.org/T437576) [14:29:10] (03CR) 10Tryvix1509: afwiki: Create Draft and Draft talk namespaces (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338902 (https://phabricator.wikimedia.org/T437576) (owner: 10Tryvix1509) [14:29:11] (03PS1) 10Marostegui: wmnet: Fix dbstore1007:x4 port [dns] - 10https://gerrit.wikimedia.org/r/1341212 (https://phabricator.wikimedia.org/T437839) [14:29:46] Lucas_WMDE: Fixed, please review. [14:29:46] I think that’s up to the deployer. While I’d also like to deploy it as originally planned, if there’s an issue with the system, we should fix it first. [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260914T1430) [14:30:10] FIRING: [2x] ProbeDown: Service sessionstore1004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:30:52] (03CR) 10Marostegui: [C:03+2] wmnet: Fix dbstore1007:x4 port [dns] - 10https://gerrit.wikimedia.org/r/1341212 (https://phabricator.wikimedia.org/T437839) (owner: 10Marostegui) [14:30:56] !log marostegui@dns1004 START - running authdns-update [14:31:14] (03CR) 10JMeybohm: [C:03+1] admin_ng: move cfssl's issuer endpoint to PKI hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341183 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [14:31:28] (03CR) 10Hashar: [C:03+1] "Looks like, it seems it was pending on a few Puppet 8 changes that were flagged via `Depends-On`." [puppet] - 10https://gerrit.wikimedia.org/r/1329284 (https://phabricator.wikimedia.org/T435917) (owner: 10Hashar) [14:32:44] (03CR) 10Hashar: "The parent change ( I605a7d6450bde982831cd67722b24b41159f1562 ) got updated to bump `test_on` to test Bookworm/Trixie. It has `Depends-On`" [puppet] - 10https://gerrit.wikimedia.org/r/1329251 (https://phabricator.wikimedia.org/T435917) (owner: 10Hashar) [14:32:54] !log eevans@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1004.eqiad.wmnet with reason: host reimage [14:33:08] !log marostegui@dns1004 END - running authdns-update [14:33:11] (03CR) 10JMeybohm: k8s: Add a script to sync kubelet labels with the API server (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1341181 (owner: 10JMeybohm) [14:35:10] RESOLVED: [2x] ProbeDown: Service sessionstore1004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:38:24] (03CR) 10Dpogorzelski: [C:03+1] liftwing-studio: add to service catalog [puppet] - 10https://gerrit.wikimedia.org/r/1341130 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [14:38:29] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 14 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338274 (https://phabricator.wikimedia.org/T432849) (owner: 10Jforrester) [14:38:33] !log eevans@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1004.eqiad.wmnet with reason: host reimage [14:38:51] (03CR) 10Dpogorzelski: [C:03+2] liftwing-studio: add to service catalog [puppet] - 10https://gerrit.wikimedia.org/r/1341130 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [14:39:55] (03CR) 10Dpogorzelski: [C:03+2] services_proxy: send SNI on the inference listener [puppet] - 10https://gerrit.wikimedia.org/r/1341163 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [14:40:10] FIRING: [2x] ProbeDown: Service sessionstore1004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:41:06] (03CR) 10Dpogorzelski: [C:03+2] liftwing-studio: chart for Open WebUI and LiteLLM [deployment-charts] - 10https://gerrit.wikimedia.org/r/1339756 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [14:42:01] 06SRE, 06Infrastructure-Foundations: Add Nodejs 26 from Nodesource to Wikimedia apt - https://phabricator.wikimedia.org/T437510#12318348 (10MoritzMuehlenhoff) p:05Triage→03Medium a:03MoritzMuehlenhoff [14:42:14] 06SRE, 06Infrastructure-Foundations: Create nodejs 26 production images - https://phabricator.wikimedia.org/T437509#12318350 (10MoritzMuehlenhoff) p:05Triage→03Medium a:03MoritzMuehlenhoff [14:42:24] FIRING: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:45:14] (03PS2) 10Dpogorzelski: liftwing-studio: chart deployment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341174 (https://phabricator.wikimedia.org/T437706) [14:45:30] (03CR) 10Dpogorzelski: liftwing-studio: chart deployment (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341174 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [14:46:19] (03CR) 10Dpogorzelski: liftwing-studio: chart deployment (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341174 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [14:48:23] (03CR) 10CoderAnimeshPathak: "Hello @mapreader@olum.org and @bcornwall@wikimedia.org," [puppet] - 10https://gerrit.wikimedia.org/r/1340598 (https://phabricator.wikimedia.org/T437799) (owner: 10CoderAnimeshPathak) [14:48:28] (03CR) 10Elukey: build-bare-slim: Avoid snapshot.debian.org by using --non-debian (WIP) (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1341110 (https://phabricator.wikimedia.org/T437829) (owner: 10Muehlenhoff) [14:50:34] (03CR) 10Federico Ceratto: [C:03+2] core-mysql.my.cnf.erb: support templating expire_logs_days for core_test [puppet] - 10https://gerrit.wikimedia.org/r/1337601 (https://phabricator.wikimedia.org/T435059) (owner: 10Federico Ceratto) [14:52:25] FIRING: [2x] SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:54:47] (03CR) 10Muehlenhoff: build-bare-slim: Avoid snapshot.debian.org by using --non-debian (WIP) (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1341110 (https://phabricator.wikimedia.org/T437829) (owner: 10Muehlenhoff) [14:55:04] (03PS2) 10Filippo Giunchedi: hieradata: organize wmcs cluster into separate cloud clusters [puppet] - 10https://gerrit.wikimedia.org/r/1338132 (https://phabricator.wikimedia.org/T437272) [14:56:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.35% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:57:24] FIRING: [3x] SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:59:15] !log eevans@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1004.eqiad.wmnet with OS bookworm [15:01:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.35% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:06:38] !log eevans@cumin1004 START - Cookbook sre.hosts.reboot-single for host sessionstore1004.eqiad.wmnet [15:07:25] FIRING: [4x] SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:08:19] (03CR) 10Giuseppe Lavagetto: [C:04-1] "Overall the approach is ok, I think we can avoid all the parsing by crafting a config file that depends on variables known to puppet other" [puppet] - 10https://gerrit.wikimedia.org/r/1341181 (owner: 10JMeybohm) [15:09:31] (03CR) 10Lucas Werkmeister (WMDE): [C:03+1] "The viwiki block still has the “older” format, but okay, at least that way it’s consistent with some other wikis. (I haven’t checked if al" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335401 (https://phabricator.wikimedia.org/T437006) (owner: 10Tryvix1509) [15:10:13] (03CR) 10Lucas Werkmeister (WMDE): [C:03+1] afwiki: Create Draft and Draft talk namespaces [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338902 (https://phabricator.wikimedia.org/T437576) (owner: 10Tryvix1509) [15:10:18] sorry, got sidetracked again [15:10:19] jouncebot: now [15:10:19] No deployments scheduled for the next 0 hour(s) and 19 minute(s) [15:10:23] jouncebot: next [15:10:23] In 0 hour(s) and 19 minute(s): Wikimedia Portals Update (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260914T1530) [15:10:30] I think that’s harmless enough in parallel [15:10:41] Hide_on_rosie: if you’re still there we could try to deploy those two changes now? [15:10:52] I'm here [15:11:00] We could try. [15:11:15] ok, let’s try [15:11:19] let’s do just one change first [15:11:24] (03CR) 10Elukey: build-bare-slim: Avoid snapshot.debian.org by using --non-debian (WIP) (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1341110 (https://phabricator.wikimedia.org/T437829) (owner: 10Muehlenhoff) [15:11:28] because if the deployment doesn’t work we might have to revert it [15:11:51] (03PS1) 10Joal: Add new field to turnilo webrequest_sampled conf [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341230 (https://phabricator.wikimedia.org/T437638) [15:12:05] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338902 (https://phabricator.wikimedia.org/T437576) (owner: 10Tryvix1509) [15:12:52] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-presto1007 - https://phabricator.wikimedia.org/T434505#12318469 (10VRiley-WMF) 05Open→03In progress Received the drive, and proceeding with replacment. Looking at the ticket history of these units, it's safe to replace without notifying the service owner. [15:13:02] (03Merged) 10jenkins-bot: afwiki: Create Draft and Draft talk namespaces [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338902 (https://phabricator.wikimedia.org/T437576) (owner: 10Tryvix1509) [15:13:15] !log lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1338902|afwiki: Create Draft and Draft talk namespaces (T437576)]] [15:13:15] (03CR) 10Clément Goubert: [C:03+1] Add new field to turnilo webrequest_sampled conf [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341230 (https://phabricator.wikimedia.org/T437638) (owner: 10Joal) [15:13:19] T437576: Add a Draft namespaces to afwiki - https://phabricator.wikimedia.org/T437576 [15:13:39] !log eevans@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sessionstore1004.eqiad.wmnet [15:14:45] image builds completed, so far so good… [15:15:10] RESOLVED: [2x] ProbeDown: Service sessionstore1004-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:17:13] (03PS1) 10Muehlenhoff: update the note for adding folks to ops [puppet] - 10https://gerrit.wikimedia.org/r/1341233 [15:17:24] FIRING: [6x] SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:17:30] !log eevans@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sessionstore1005.eqiad.wmnet with reason: Firmware upgrades — T437516 [15:17:31] !log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, tryvix1509: Backport for [[gerrit:1338902|afwiki: Create Draft and Draft talk namespaces (T437576)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:17:33] T437516: sessionstore: Debian Bookworm upgrades - https://phabricator.wikimedia.org/T437516 [15:17:44] !log eevans@cumin1004 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts sessionstore1005.eqiad.wmnet [15:17:51] (03PS1) 10Elukey: profile::pki::client: move the signer endpoint to pki[12]002 [puppet] - 10https://gerrit.wikimedia.org/r/1341234 (https://phabricator.wikimedia.org/T436809) [15:17:51] Hide_on_rosie: please test! [15:18:27] (03CR) 10CI reject: [V:04-1] profile::pki::client: move the signer endpoint to pki[12]002 [puppet] - 10https://gerrit.wikimedia.org/r/1341234 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [15:19:10] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware), 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12318484 (10fgiunchedi) a:05fgiunchedi→03Jclark-ctr >>! In T419892#12317656, @Andrew wrote: >>>! In T419892#12317356, @fgiunchedi... [15:19:17] (03PS2) 10Elukey: profile::pki::client: move the signer endpoint to pki[12]002 [puppet] - 10https://gerrit.wikimedia.org/r/1341234 (https://phabricator.wikimedia.org/T436809) [15:19:45] (03PS3) 10Elukey: profile::pki::client: move the signer endpoint to pki[12]002 [puppet] - 10https://gerrit.wikimedia.org/r/1341234 (https://phabricator.wikimedia.org/T436809) [15:19:52] (03CR) 10CI reject: [V:04-1] profile::pki::client: move the signer endpoint to pki[12]002 [puppet] - 10https://gerrit.wikimedia.org/r/1341234 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [15:20:29] (03CR) 10CI reject: [V:04-1] profile::pki::client: move the signer endpoint to pki[12]002 [puppet] - 10https://gerrit.wikimedia.org/r/1341234 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [15:20:54] https://usercontent.irccloud-cdn.com/file/ogBBhjx3/image.png [15:20:55] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [15:21:00] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 15 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340505 (https://phabricator.wikimedia.org/T437772) (owner: 10Bunnypranav) [15:21:03] Lucas_WMDE: afwiki check OK https://af.wikipedia.org/w/index.php?title=Konsep:T437576&action=info [15:21:04] T437576: Add a Draft namespaces to afwiki - https://phabricator.wikimedia.org/T437576 [15:21:20] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [15:21:37] Hey Lucas_WMDE can you do mine as well? [15:21:52] !log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, tryvix1509: Continuing with deployment [15:21:56] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-presto1007 - https://phabricator.wikimedia.org/T434505#12318491 (10VRiley-WMF) 05In progress→03Open Swapped drive and it is currently rebuilding. Will close this ticket when it completes rebuilding. [15:22:23] (03PS4) 10Elukey: profile::pki::client: move the signer endpoint to pki[12]002 [puppet] - 10https://gerrit.wikimedia.org/r/1341234 (https://phabricator.wikimedia.org/T436809) [15:22:30] How do I test viwiki EC status? [15:22:34] bunnypranav: maybe, yes [15:22:37] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply [15:22:52] Hide_on_rosie: I suspect that can’t be tested beyond “wiki isn’t broken” [15:22:56] I would appreciate, it is a very small config change. https://gerrit.wikimedia.org/r/c/1340505/ [15:22:57] (but also, it’s not deployed yet) [15:23:10] okay. [15:23:11] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply [15:23:22] (03CR) 10Elukey: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1341234 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [15:23:32] jan_drewniak_: is it okay if I deploy some more config changes during the Wikimedia Portals Update window? [15:24:27] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply [15:24:59] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply [15:25:05] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply [15:25:37] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply [15:25:58] (03CR) 10Giuseppe Lavagetto: [C:03+1] update the note for adding folks to ops [puppet] - 10https://gerrit.wikimedia.org/r/1341233 (owner: 10Muehlenhoff) [15:26:13] (03CR) 10Brouberol: [C:03+1] Add new field to turnilo webrequest_sampled conf [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341230 (https://phabricator.wikimedia.org/T437638) (owner: 10Joal) [15:26:31] (03CR) 10Brouberol: [C:03+2] Add new field to turnilo webrequest_sampled conf [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341230 (https://phabricator.wikimedia.org/T437638) (owner: 10Joal) [15:26:51] (03PS5) 10Elukey: profile::pki::client: move the signer endpoint to pki[12]002 [puppet] - 10https://gerrit.wikimedia.org/r/1341234 (https://phabricator.wikimedia.org/T436809) [15:27:35] (03CR) 10Elukey: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1341234 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [15:28:16] (03CR) 10Muehlenhoff: [C:03+2] update the note for adding folks to ops [puppet] - 10https://gerrit.wikimedia.org/r/1341233 (owner: 10Muehlenhoff) [15:28:20] (03CR) 10Elukey: profile::pki::client: move the signer endpoint to pki[12]002 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1341234 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [15:28:59] !log lucaswerkmeister-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1338902|afwiki: Create Draft and Draft talk namespaces (T437576)]] (duration: 15m 44s) [15:29:03] T437576: Add a Draft namespaces to afwiki - https://phabricator.wikimedia.org/T437576 [15:29:46] !log lucaswerkmeister-wmde@deploy1003 mwscript-k8s job started: namespaceDupes afwiki --fix # T437576 [15:30:05] jan_drewniak: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for Wikimedia Portals Update. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260914T1530). [15:30:20] (03CR) 10Lucas Werkmeister (WMDE): [C:03+1] core-Namespaces: Add AW and AWT alias for its talk in abstractwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340505 (https://phabricator.wikimedia.org/T437772) (owner: 10Bunnypranav) [15:30:50] I’ll wait a few minutes to see if it’s okay to deploy during this window or not [15:30:56] sure [15:30:58] eevans@cumin1004 upgrade-firmware (PID 2149967) is awaiting input [15:31:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.93% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:31:29] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review, 06Release-Engineering-Team (Radar): apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12318562 (10Raymond_Ndibe) We noticed this problem today while deploy to toolforge k8s cluster. Builds... [15:32:09] please do ping me when you are ready :) [15:32:27] !log eevans@cumin1004 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts sessionstore1005.eqiad.wmnet [15:32:53] (03CR) 10Ahmon Dancy: [C:03+1] "LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/1340987 (https://phabricator.wikimedia.org/T437069) (owner: 10Muehlenhoff) [15:33:20] (03CR) 10Elukey: [C:03+2] Add pki svc IP A and PTR records [dns] - 10https://gerrit.wikimedia.org/r/1338234 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [15:33:38] !log eevans@cumin1004 START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:33:48] !log elukey@dns1004 START - running authdns-update [15:34:14] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.13 point update - https://phabricator.wikimedia.org/T414205#12318576 (10MoritzMuehlenhoff) [15:34:16] !log eevans@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:34:33] (03PS1) 10DCausse: search: add semantic-highlighter to liftwing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341239 (https://phabricator.wikimedia.org/T433872) [15:35:01] !log joal@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply [15:35:25] eh, let’s continue deploying Hide_on_rosie bunnypranav [15:35:36] !log installing shadow security updates [15:35:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:35:40] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335401 (https://phabricator.wikimedia.org/T437006) (owner: 10Tryvix1509) [15:35:40] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340505 (https://phabricator.wikimedia.org/T437772) (owner: 10Bunnypranav) [15:35:45] thank! [15:35:57] !log joal@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply [15:36:00] !log eevans@cumin1004 START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bookworm [15:36:04] !log elukey@dns1004 END - running authdns-update [15:37:10] (03Merged) 10jenkins-bot: Adjust extendedconfirmed calculation to first edit on viwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335401 (https://phabricator.wikimedia.org/T437006) (owner: 10Tryvix1509) [15:39:11] (03Merged) 10jenkins-bot: core-Namespaces: Add AW and AWT alias for its talk in abstractwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340505 (https://phabricator.wikimedia.org/T437772) (owner: 10Bunnypranav) [15:39:25] !log lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1335401|Adjust extendedconfirmed calculation to first edit on viwiki (T437006)]], [[gerrit:1340505|core-Namespaces: Add AW and AWT alias for its talk in abstractwiki (T437772)]] [15:39:30] T437006: Adjust extendedconfirmed calculation to first edit on viwiki - https://phabricator.wikimedia.org/T437006 [15:39:30] T437772: Add "AW" as a namespace alias for the "Abstract Wikipedia:" project namespace on abstractwiki - https://phabricator.wikimedia.org/T437772 [15:39:40] FIRING: [2x] ProbeDown: Service sessionstore1005-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:41:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.55% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:43:30] !log lucaswerkmeister-wmde@deploy1003 bunnypranav, lucaswerkmeister-wmde, tryvix1509: Backport for [[gerrit:1335401|Adjust extendedconfirmed calculation to first edit on viwiki (T437006)]], [[gerrit:1340505|core-Namespaces: Add AW and AWT alias for its talk in abstractwiki (T437772)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:43:44] bunnypranav: please test! [15:43:52] Hide_on_rosie: please check that viwiki is still working ^^ [15:43:54] (03PS1) 10STran: SuggestedInvestigations: Add and enable 'sockpuppets' queue view [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341242 (https://phabricator.wikimedia.org/T437183) [15:44:21] looks good! [15:44:40] Check OK! [15:45:00] (03CR) 10CI reject: [V:04-1] SuggestedInvestigations: Add and enable 'sockpuppets' queue view [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341242 (https://phabricator.wikimedia.org/T437183) (owner: 10STran) [15:45:21] !log lucaswerkmeister-wmde@deploy1003 bunnypranav, lucaswerkmeister-wmde, tryvix1509: Continuing with deployment [15:45:23] thanks! [15:47:17] (03CR) 10Dreamy Jazz: SuggestedInvestigations: Add and enable 'sockpuppets' queue view (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341242 (https://phabricator.wikimedia.org/T437183) (owner: 10STran) [15:47:28] (03CR) 10Dreamy Jazz: SuggestedInvestigations: Add and enable 'sockpuppets' queue view (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341242 (https://phabricator.wikimedia.org/T437183) (owner: 10STran) [15:47:32] !log eevans@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bookworm [15:48:03] (03CR) 10Dreamy Jazz: SuggestedInvestigations: Add and enable 'sockpuppets' queue view (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341242 (https://phabricator.wikimedia.org/T437183) (owner: 10STran) [15:48:16] !log eevans@cumin1004 START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bookworm [15:48:33] (03PS2) 10STran: SuggestedInvestigations: Add and enable 'sockpuppets' queue view [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341242 (https://phabricator.wikimedia.org/T437183) [15:48:36] (03CR) 10STran: SuggestedInvestigations: Add and enable 'sockpuppets' queue view (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341242 (https://phabricator.wikimedia.org/T437183) (owner: 10STran) [15:49:12] (03PS3) 10STran: SuggestedInvestigations: Add and enable 'sockpuppets' queue view [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341242 (https://phabricator.wikimedia.org/T437183) [15:49:37] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:49:40] RESOLVED: [2x] ProbeDown: Service sessionstore1005-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:49:48] !log lucaswerkmeister-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1335401|Adjust extendedconfirmed calculation to first edit on viwiki (T437006)]], [[gerrit:1340505|core-Namespaces: Add AW and AWT alias for its talk in abstractwiki (T437772)]] (duration: 10m 23s) [15:49:53] T437006: Adjust extendedconfirmed calculation to first edit on viwiki - https://phabricator.wikimedia.org/T437006 [15:49:53] T437772: Add "AW" as a namespace alias for the "Abstract Wikipedia:" project namespace on abstractwiki - https://phabricator.wikimedia.org/T437772 [15:49:58] (03CR) 10STran: SuggestedInvestigations: Add and enable 'sockpuppets' queue view (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341242 (https://phabricator.wikimedia.org/T437183) (owner: 10STran) [15:50:08] (03CR) 10CI reject: [V:04-1] SuggestedInvestigations: Add and enable 'sockpuppets' queue view [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341242 (https://phabricator.wikimedia.org/T437183) (owner: 10STran) [15:50:11] !log lucaswerkmeister-wmde@deploy1003 mwscript-k8s job started: namespaceDupes abstractwiki --fix # T437772 [15:50:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.85% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:50:36] That a lot for the deploy! [15:52:24] FIRING: [6x] SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:53:14] I created my token so this should resolve soon? ^ [15:53:23] (03PS4) 10STran: SuggestedInvestigations: Add and enable 'sockpuppets' queue view [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341242 (https://phabricator.wikimedia.org/T437183) [15:53:29] !log UTC afternoon backport+config window belatedly done [15:53:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:53:53] (03PS1) 10Elukey: docker_registry::instance: proper restart when the config changes [puppet] - 10https://gerrit.wikimedia.org/r/1341249 (https://phabricator.wikimedia.org/T428555) [15:53:54] 06SRE, 06Collaboration-Services, 10Phabricator, 06Traffic, 13Patch-For-Review: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228#12318700 (10LSobanski) 05Open→03Declined We're not planning to work on this task. [15:54:02] fyi tgr_ most of the changes are deployed now (stephanebisson’s and yours are still left) [15:55:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.72% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:57:25] FIRING: [6x] SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:58:20] (03CR) 10Dreamy Jazz: SuggestedInvestigations: Add and enable 'sockpuppets' queue view (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341242 (https://phabricator.wikimedia.org/T437183) (owner: 10STran) [15:58:52] 10ops-eqiad, 10Cassandra, 06DC-Ops: sessionstore1005.eqiad.wmnet is down (again) - https://phabricator.wikimedia.org/T437915#12318752 (10Eevans) p:05Triage→03Unbreak! [16:03:41] (03PS5) 10STran: SuggestedInvestigations: Add and enable 'sockpuppets' queue view [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341242 (https://phabricator.wikimedia.org/T437183) [16:05:25] !log drop non-links tables from db2247 (T437278) [16:05:27] (03CR) 10Ozge: search: add semantic-highlighter to liftwing (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341239 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [16:05:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:05:30] T437278: Drop unneeded tables from x4 and s4 - https://phabricator.wikimedia.org/T437278 [16:05:35] (03PS1) 10STran: SI: Implement "queue view" data functionality [extensions/CheckUser] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1341253 (https://phabricator.wikimedia.org/T437183) [16:05:48] (03PS1) 10STran: Add wmf-specific Special:SuggestedInvestigations messages [extensions/WikimediaMessages] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1341254 (https://phabricator.wikimedia.org/T437183) [16:07:20] !log drop links tables from db2240 (T437278) [16:07:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:07:25] FIRING: [6x] SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:08:34] (03CR) 10Dreamy Jazz: SuggestedInvestigations: Add and enable 'sockpuppets' queue view (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341242 (https://phabricator.wikimedia.org/T437183) (owner: 10STran) [16:08:36] (03CR) 10DCausse: search: add semantic-highlighter to liftwing (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341239 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [16:09:32] !log drop links tables from db1252 (T437278) [16:09:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:11:40] (03PS2) 10DCausse: search: add semantic-highlighter-staging to liftwing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341239 (https://phabricator.wikimedia.org/T433872) [16:11:40] (03PS1) 10DCausse: search: add semantic-highlighter to liftwing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341259 (https://phabricator.wikimedia.org/T433872) [16:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:12:25] FIRING: [5x] SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:14:37] (03CR) 10Ozge: [C:03+1] search: add semantic-highlighter-staging to liftwing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341239 (https://phabricator.wikimedia.org/T433872) (owner: 10DCausse) [16:15:22] (03PS2) 10Federico Ceratto: core-mysql.my.cnf.erb: add role:: in role name syntax [puppet] - 10https://gerrit.wikimedia.org/r/1341257 (https://phabricator.wikimedia.org/T435059) [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:17:24] FIRING: [4x] SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:21:14] (03PS6) 10STran: SuggestedInvestigations: Add and enable 'sockpuppets' queue view [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341242 (https://phabricator.wikimedia.org/T437183) [16:22:02] (03CR) 10Federico Ceratto: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1341257 (https://phabricator.wikimedia.org/T435059) (owner: 10Federico Ceratto) [16:22:58] (03Abandoned) 10Dzahn: varnish: new policy to allow websockets and caching, apply to phab [puppet] - 10https://gerrit.wikimedia.org/r/1171263 (https://phabricator.wikimedia.org/T274228) (owner: 10Dzahn) [16:23:29] (03CR) 10Dzahn: "thanks for this" [puppet] - 10https://gerrit.wikimedia.org/r/1340506 (owner: 10Elukey) [16:24:10] (03CR) 10Dzahn: "also affected contint hosts - thanks for fix" [puppet] - 10https://gerrit.wikimedia.org/r/1340942 (owner: 10Giuseppe Lavagetto) [16:25:23] (03Abandoned) 10Ladsgroup: Run tests on CI [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337608 (owner: 10Ladsgroup) [16:25:35] (03CR) 10Ladsgroup: "recheck" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1339064 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup-claude) [16:26:16] (03CR) 10STran: SuggestedInvestigations: Add and enable 'sockpuppets' queue view (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341242 (https://phabricator.wikimedia.org/T437183) (owner: 10STran) [16:27:45] 06SRE, 06Traffic, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12318925 (10ssingh) @BCornwall can take care of this from Traffic, for input and the patch. [16:28:28] (03PS1) 10Subramanya Sastry: Parsoid Read Views: Enable on all namespaces on wikitech (labswiki) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341271 (https://phabricator.wikimedia.org/T437916) [16:28:32] 10ops-eqiad, 10Cassandra, 06DC-Ops: sessionstore1005.eqiad.wmnet is down (again) - https://phabricator.wikimedia.org/T437915#12318928 (10Jclark-ctr) Updating firmware on bios / raid controller / backplane @Eevans Server is now showing resolved ` A configuration related issue on the device Backplane i... [16:33:18] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review, 06Release-Engineering-Team (Radar): apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12318952 (10elukey) The base images are being rebuilt as we speak and from some preliminary tests the sn... [16:33:33] 06SRE, 06Traffic, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12318955 (10Dzahn) Appreciate the comment from Pppery here. It's not always a big formal decision; a lot of times it's just a few people t... [16:35:57] 10ops-eqiad, 06SRE, 10Cassandra, 06DC-Ops: sessionstore1005.eqiad.wmnet is down (again) - https://phabricator.wikimedia.org/T437915#12318988 (10Jclark-ctr) a:03Jclark-ctr [16:36:19] (03CR) 10BCornwall: [C:03+2] ncredir: Fix wikipedia.ar redirect to Spanish Wikipedia [puppet] - 10https://gerrit.wikimedia.org/r/1340598 (https://phabricator.wikimedia.org/T437799) (owner: 10CoderAnimeshPathak) [16:42:25] RESOLVED: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:43:10] (03CR) 10TrainBranchBot: [C:03+2] "Approved by tgr@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1330446 (https://phabricator.wikimedia.org/T419684) (owner: 10Arendpieter) [16:44:20] (03Merged) 10jenkins-bot: CommonSettings: Use a restrictive, eval-free CSP for auth.wikimedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1330446 (https://phabricator.wikimedia.org/T419684) (owner: 10Arendpieter) [16:44:34] !log tgr@deploy1003 Started scap sync-world: Backport for [[gerrit:1330446|CommonSettings: Use a restrictive, eval-free CSP for auth.wikimedia.org (T419684)]] [16:44:37] T419684: Add restrictive CSP to auth.wikimedia.org - https://phabricator.wikimedia.org/T419684 [16:44:38] 06SRE, 06Traffic, 10WMF-General-or-Unknown, 13Patch-For-Review: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12319023 (10BCornwall) 05Open→03Resolved Thanks for reporting and for the patch! Merged, and now appropriately redirecting to es.wp.o [16:46:10] (03PS1) 10MusikAnimal: [CodeMirror] enable for new users (enwiki), new VE integration (global) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341273 (https://phabricator.wikimedia.org/T288161) [16:47:57] (03CR) 10Dreamy Jazz: [C:03+1] SuggestedInvestigations: Add and enable 'sockpuppets' queue view [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341242 (https://phabricator.wikimedia.org/T437183) (owner: 10STran) [16:48:41] (03PS1) 10STran: SI: Implement "queue view" functionality [extensions/CheckUser] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1341274 (https://phabricator.wikimedia.org/T437183) [16:49:04] !log tgr@deploy1003 arendpieter, tgr: Backport for [[gerrit:1330446|CommonSettings: Use a restrictive, eval-free CSP for auth.wikimedia.org (T419684)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:50:17] (03CR) 10Dzahn: "with the backup just taking 15 minutes I would also tend to not skip it anymore" [cookbooks] - 10https://gerrit.wikimedia.org/r/1339602 (https://phabricator.wikimedia.org/T437680) (owner: 10Arnaudb) [16:51:26] (03PS2) 10STran: SI: Implement "queue view" functionality [extensions/CheckUser] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1341274 (https://phabricator.wikimedia.org/T437183) [16:51:48] (03CR) 10Dzahn: [C:03+1] "makes sense to me (though I have not actually verified the nginx config)" [puppet] - 10https://gerrit.wikimedia.org/r/1341184 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [16:51:55] !log eevans@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bookworm [16:52:12] !log eevans@cumin1004 START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bookworm [16:56:21] (03CR) 10Dzahn: "this fixed even more things like https://phabricator.wikimedia.org/T437797 :)" [puppet] - 10https://gerrit.wikimedia.org/r/1340942 (owner: 10Giuseppe Lavagetto) [17:00:00] (03CR) 10Dreamy Jazz: [C:03+1] "Agreed as a team that we want to backport these changes to wmf.19" [extensions/CheckUser] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1341274 (https://phabricator.wikimedia.org/T437183) (owner: 10STran) [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260914T1700) [17:00:05] ryankemper: How many deployers does it take to do Wikidata Query Service weekly deploy deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260914T1700). [17:00:14] (03CR) 10Gergő Tisza: [C:03+1] "Fails with" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1330446 (https://phabricator.wikimedia.org/T419684) (owner: 10Arendpieter) [17:00:35] !log tgr@deploy1003 arendpieter, tgr: Rolling back deployment [17:03:04] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 14 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339741 (https://phabricator.wikimedia.org/T436199) (owner: 10LWatson) [17:04:22] (03CR) 10BCornwall: [C:03+1] dns: add liftwing-studio records [dns] - 10https://gerrit.wikimedia.org/r/1341128 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [17:05:04] (03PS1) 10Gergő Tisza: Revert "CommonSettings: Use a restrictive, eval-free CSP for auth.wikimedia.org" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341276 [17:05:04] (03PS3) 10Samtar: extension-list: Add ModeratorToolkit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337611 (https://phabricator.wikimedia.org/T431000) [17:05:04] (03CR) 10Samtar: "This is now ready for review & merge" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337611 (https://phabricator.wikimedia.org/T431000) (owner: 10Samtar) [17:05:27] (03PS4) 10Samtar: extension-list: Add ModeratorToolkit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337611 (https://phabricator.wikimedia.org/T431000) [17:06:25] (03CR) 10Gergő Tisza: [C:03+2] Revert "CommonSettings: Use a restrictive, eval-free CSP for auth.wikimedia.org" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341276 (owner: 10Gergő Tisza) [17:06:26] (03CR) 10BCornwall: [V:03+1] "PCC SUCCESS (DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9414/console" [puppet] - 10https://gerrit.wikimedia.org/r/1298885 (owner: 10BCornwall) [17:07:30] (03Merged) 10jenkins-bot: Revert "CommonSettings: Use a restrictive, eval-free CSP for auth.wikimedia.org" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341276 (owner: 10Gergő Tisza) [17:07:54] !log tgr@deploy1003 Finished scap sync-world: Backport for [[gerrit:1330446|CommonSettings: Use a restrictive, eval-free CSP for auth.wikimedia.org (T419684)]] (duration: 23m 19s) [17:07:57] T419684: Add restrictive CSP to auth.wikimedia.org - https://phabricator.wikimedia.org/T419684 [17:09:25] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 15 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/CheckUser] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1341274 (https://phabricator.wikimedia.org/T437183) (owner: 10STran) [17:09:28] (03PS1) 10Jdlrobson: [A11y] Add list semantics to ReadingList page [extensions/ReadingLists] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1341278 (https://phabricator.wikimedia.org/T435864) [17:09:39] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 15 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341242 (https://phabricator.wikimedia.org/T437183) (owner: 10STran) [17:10:09] (03CR) 10CDanis: "FYI" [puppet] - 10https://gerrit.wikimedia.org/r/1327104 (https://phabricator.wikimedia.org/T427465) (owner: 10Ladsgroup) [17:14:07] (03PS2) 10Jdlrobson: [A11y] Add list semantics to ReadingList page [extensions/ReadingLists] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1341278 (https://phabricator.wikimedia.org/T435864) [17:16:53] (03PS5) 10Samtar: extension-list: Add ModeratorToolkit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337611 (https://phabricator.wikimedia.org/T431000) [17:20:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [17:36:06] eevans@cumin1004 reimage (PID 2245588) is awaiting input [17:36:43] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to releasers-mobile for LPetty-WMF - https://phabricator.wikimedia.org/T437662#12319271 (10Dzahn) a:03thcipriani [17:37:04] (03PS1) 10Arendpieter: CommonSettings: Use a restrictive CSP for auth.wikimedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341292 (https://phabricator.wikimedia.org/T419684) [17:37:41] 06SRE, 10SRE-Access-Requests: Requesting access to deployment and restricted for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12319280 (10Dzahn) Hi @thcipriani a deployment request for your consideration. [17:38:40] 06SRE, 10SRE-Access-Requests: Requesting access to deployment and restricted for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12319282 (10Dzahn) Hi @EAlbizzati-WMF please get your manager to approve this request here on ticket. Also, Urbanecm is right, you just need deployment and it will include... [17:41:11] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to growthbook-customelevatedaccess - https://phabricator.wikimedia.org/T437611#12319303 (10Dzahn) a:03AKanji-WMF Hi Anil the patch for this, https://gerrit.wikimedia.org/r/c/operations/puppet/+/1339129 seems all good to go and all boxes he... [17:41:22] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 14 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [extensions/ReadingLists] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1341278 (https://phabricator.wikimedia.org/T435864) (owner: 10Jdlrobson) [17:42:06] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users, growthbook-customelevatedaccess for Anil - https://phabricator.wikimedia.org/T437611#12319305 (10Dzahn) [17:42:29] (03CR) 10Ahmon Dancy: [C:03+1] "LGTM!" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1339635 (https://phabricator.wikimedia.org/T416452) (owner: 10Muehlenhoff) [17:43:01] 10SRE-Access-Requests, 06Infrastructure-Foundations, 10LDAP-Access-Requests, 13Patch-For-Review: Superset data access request for abibendall - https://phabricator.wikimedia.org/T430938#12319307 (10Dzahn) Hi @ABendall-WMF this is still waiting for your input. Cheers [17:43:56] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment and restricted for cooltey - https://phabricator.wikimedia.org/T437658#12319308 (10Dzahn) a:03thcipriani [17:44:14] (03PS1) 10Jdlrobson: Enable ReadingLists for all logged-in users on English Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338999 (https://phabricator.wikimedia.org/T434923) [17:44:41] (03CR) 10Majavah: [C:03+1] extension-list: Add ModeratorToolkit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337611 (https://phabricator.wikimedia.org/T431000) (owner: 10Samtar) [17:45:23] (03PS1) 10Jdlrobson: Enable experiment on test wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340004 (https://phabricator.wikimedia.org/T437665) [17:45:26] (03PS2) 10Jdlrobson: Enable experiment on test wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340004 (https://phabricator.wikimedia.org/T437665) [17:45:48] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment and analytics-privatedata-users for Cklimas - https://phabricator.wikimedia.org/T437653#12319310 (10Dzahn) > The server I'm aware we use for this is deployment.eqiad.wmnet deployment.eqiad.wmnet is an alias name that internal... [17:46:10] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users level 3 for derenrich - https://phabricator.wikimedia.org/T437668#12319311 (10ssingh) [17:46:26] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 14 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338999 (https://phabricator.wikimedia.org/T434923) (owner: 10Jdlrobson) [17:46:52] jclark@cumin1003 provision (PID 34205) is awaiting input [17:48:09] !log jclark@cumin1003 START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:50:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [17:50:40] FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker1383:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1383 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [17:53:21] (03PS3) 10Jdlrobson: Enable Reading Recommendations experiment on test wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340004 (https://phabricator.wikimedia.org/T437665) [17:54:13] (03PS1) 10Majavah: O:wmcs: Add new cloudinfra_etcd role [puppet] - 10https://gerrit.wikimedia.org/r/1341298 (https://phabricator.wikimedia.org/T410721) [17:54:30] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 14 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340004 (https://phabricator.wikimedia.org/T437665) (owner: 10Jdlrobson) [17:55:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker1383:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1383 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [17:57:28] jouncebot: nowandnext [17:57:28] For the next 0 hour(s) and 2 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260914T1700) [17:57:28] In 2 hour(s) and 2 minute(s): UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260914T2000) [17:57:48] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (NOOP 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9415/console" [puppet] - 10https://gerrit.wikimedia.org/r/1341298 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [17:59:01] (03CR) 10TrainBranchBot: [C:03+2] "Approved by samtar@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337611 (https://phabricator.wikimedia.org/T431000) (owner: 10Samtar) [18:00:58] (03Merged) 10jenkins-bot: extension-list: Add ModeratorToolkit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1337611 (https://phabricator.wikimedia.org/T431000) (owner: 10Samtar) [18:01:22] !log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [18:03:19] tgr_: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1341276 is merged but not deployed - just had a warning for it while doing a deploy. Is it okay to push out? [18:03:34] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12319368 (10EAlbizzati-WMF) [18:05:11] !log samtar@deploy1003 Started scap sync-world: Backport for [[gerrit:1337611|extension-list: Add ModeratorToolkit (T431000)]] [18:05:14] T431000: Deploy the ModeratorToolkit extension to Beta Cluster - https://phabricator.wikimedia.org/T431000 [18:05:22] I am going to deploy the revert too [18:06:01] !log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bookworm [18:06:13] 10ops-eqiad, 06SRE, 10Cassandra, 06DC-Ops: sessionstore1005.eqiad.wmnet is down (again) - https://phabricator.wikimedia.org/T437915#12319389 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host sessionstore1005.eqiad.wmnet with OS bookworm [18:07:29] (03CR) 10Samtar: "This was merged but not deployed - I deployed it as part of my deployment of https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341276 (owner: 10Gergő Tisza) [18:08:05] (03PS10) 10BCornwall: varnish: Remove reload_vcl_opts function [puppet] - 10https://gerrit.wikimedia.org/r/1298885 [18:09:15] !log samtar@deploy1003 samtar: Backport for [[gerrit:1337611|extension-list: Add ModeratorToolkit (T431000)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [18:09:27] (03CR) 10BCornwall: [V:03+1] "PCC SUCCESS (DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9416/console" [puppet] - 10https://gerrit.wikimedia.org/r/1298885 (owner: 10BCornwall) [18:10:00] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12319405 (10Seddon) Approved [18:10:09] !log samtar@deploy1003 samtar: Continuing with deployment [18:11:19] (03PS11) 10BCornwall: varnish: Remove reload_vcl_opts function [puppet] - 10https://gerrit.wikimedia.org/r/1298885 [18:13:11] (03CR) 10BCornwall: "On re-review I noticed that the `-s` flag would have been broken - the new logic would pass `-s file1 -s file2` while the old logic would " [puppet] - 10https://gerrit.wikimedia.org/r/1298885 (owner: 10BCornwall) [18:14:45] !log samtar@deploy1003 Finished scap sync-world: Backport for [[gerrit:1337611|extension-list: Add ModeratorToolkit (T431000)]] (duration: 09m 34s) [18:14:49] T431000: Deploy the ModeratorToolkit extension to Beta Cluster - https://phabricator.wikimedia.org/T431000 [18:16:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [18:16:50] (03PS12) 10BCornwall: varnish: Remove reload_vcl_opts function [puppet] - 10https://gerrit.wikimedia.org/r/1298885 [18:17:51] !log jclark@cumin1003 START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [18:18:05] (03PS13) 10BCornwall: varnish: Remove reload_vcl_opts function [puppet] - 10https://gerrit.wikimedia.org/r/1298885 [18:19:22] (03CR) 10Ahmon Dancy: [C:03+1] Allow registryctl to use a custom Docker config.json filepath [docker-images/docker-report] - 10https://gerrit.wikimedia.org/r/1339621 (https://phabricator.wikimedia.org/T437297) (owner: 10Elukey) [18:19:27] (03CR) 10BCornwall: [V:03+1] "PCC SUCCESS (DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9417/console" [puppet] - 10https://gerrit.wikimedia.org/r/1298885 (owner: 10BCornwall) [18:19:57] (03PS14) 10BCornwall: varnish: Remove reload_vcl_opts function [puppet] - 10https://gerrit.wikimedia.org/r/1298885 [18:22:22] (03PS3) 10Samtar: IS/IS-labs: Set wmgUseModeratorToolkit default false [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341284 (https://phabricator.wikimedia.org/T431000) [18:22:27] (03PS4) 10Samtar: IS/IS-labs: Set wmgUseModeratorToolkit default false [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341284 (https://phabricator.wikimedia.org/T431000) [18:23:30] (03PS2) 10Majavah: O:wmcs: Add new cloudinfra_etcd role [puppet] - 10https://gerrit.wikimedia.org/r/1341298 (https://phabricator.wikimedia.org/T410721) [18:23:43] (03PS3) 10Majavah: O:wmcs: Add new cloudinfra_etcd role [puppet] - 10https://gerrit.wikimedia.org/r/1341298 (https://phabricator.wikimedia.org/T410721) [18:25:25] !log root@cumin2003 START - Cookbook sre.network.tls for network device lsw1-d2-eqiad [18:25:25] !log root@cumin2003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d2-eqiad [18:25:48] !log brett@cumin2003 START - Cookbook sre.network.tls for network device lsw1-d4-eqiad [18:25:48] !log brett@cumin2003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d4-eqiad [18:26:06] !log brett@cumin2003 START - Cookbook sre.network.tls for network device ssw1-d8-eqiad [18:26:06] !log brett@cumin2003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-d8-eqiad [18:26:13] !log brett@cumin2003 START - Cookbook sre.network.tls for network device lsw1-d6-eqiad [18:26:13] !log brett@cumin2003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-eqiad [18:41:00] (03CR) 10BCornwall: "I don't know that I agree with this approach. We need these mmdb files for sane operation of HAProxy. If they're not there, then we need t" [puppet] - 10https://gerrit.wikimedia.org/r/1224897 (https://phabricator.wikimedia.org/T414111) (owner: 10Slyngshede) [18:43:15] jclark@cumin1003 reimage (PID 37141) is awaiting input [18:45:21] jclark@cumin1003 provision (PID 37835) is awaiting input [18:48:35] (03CR) 10VolkerE: Enable Reading Recommendations experiment on test wiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340004 (https://phabricator.wikimedia.org/T437665) (owner: 10Jdlrobson) [18:49:58] !log eevans@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sessionstore1005.eqiad.wmnet with OS bookworm [18:50:06] !log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sessionstore1005.eqiad.wmnet with OS bookworm [18:50:12] 10ops-eqiad, 06SRE, 10Cassandra, 06DC-Ops: sessionstore1005.eqiad.wmnet is down (again) - https://phabricator.wikimedia.org/T437915#12319518 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host sessionstore1005.eqiad.wmnet with OS bookworm executed with errors: -... [18:50:21] !log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bookworm [18:50:29] 10ops-eqiad, 06SRE, 10Cassandra, 06DC-Ops: sessionstore1005.eqiad.wmnet is down (again) - https://phabricator.wikimedia.org/T437915#12319521 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host sessionstore1005.eqiad.wmnet with OS bookworm [18:53:12] !log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [19:00:31] (03CR) 10Dillon: [C:03+1] "LGTM, thanks!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341284 (https://phabricator.wikimedia.org/T431000) (owner: 10Samtar) [19:01:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [19:01:50] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Attempt to use a VM as Kerberos replica - https://phabricator.wikimedia.org/T435873#12319552 (10bking) 05In progress→03Resolved We took a bit of a left turn with the rolling restart cookbook (see linked ticket), but we got... [19:03:15] (03PS1) 10Southparkfan: changeprop: Switch to new jobrunner for Beta Cluster [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341323 (https://phabricator.wikimedia.org/T435393) [19:03:16] (03CR) 10TrainBranchBot: [C:03+2] "Approved by musikanimal@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341273 (https://phabricator.wikimedia.org/T288161) (owner: 10MusikAnimal) [19:04:18] (03Merged) 10jenkins-bot: [CodeMirror] enable for new users (enwiki), new VE integration (global) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1341273 (https://phabricator.wikimedia.org/T288161) (owner: 10MusikAnimal) [19:04:34] !log musikanimal@deploy1003 Started scap sync-world: Backport for [[gerrit:1341273|[CodeMirror] enable for new users (enwiki), new VE integration (global) (T288161 T432558)]] [19:04:38] T288161: Enable CodeMirror's Syntax Highlighting on English Wikipedia by default - https://phabricator.wikimedia.org/T288161 [19:04:39] T432558: The 2017 wikitext editor performs very poorly when CM6 is enabled - https://phabricator.wikimedia.org/T432558 [19:09:44] (03Abandoned) 10BCornwall: interface: Don't touch interfaces file if absent [puppet] - 10https://gerrit.wikimedia.org/r/1332813 (owner: 10BCornwall) [19:10:40] (03CR) 10BCornwall: [C:03+1] create attribution.wikimedia.org for future miscweb static site [dns] - 10https://gerrit.wikimedia.org/r/1339986 (https://phabricator.wikimedia.org/T437635) (owner: 10Dzahn) [19:12:12] (03PS1) 10JHathaway: WIP: Move DB backups from cumin1003 to cumin1004 [puppet] - 10https://gerrit.wikimedia.org/r/1341326 (https://phabricator.wikimedia.org/T427897) [19:12:38] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1341326 (https://phabricator.wikimedia.org/T427897) (owner: 10JHathaway) [19:14:20] (03CR) 10CI reject: [V:04-1] WIP: Move DB backups from cumin1003 to cumin1004 [puppet] - 10https://gerrit.wikimedia.org/r/1341326 (https://phabricator.wikimedia.org/T427897) (owner: 10JHathaway) [19:16:24] FIRING: SystemdUnitFailed: production-images-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:18:23] (03CR) 10Dzahn: [C:03+1] "thanks! so yea, a change here is not expensive and it seems like there is no blocker or principal concern about this site.. so going ahead" [dns] - 10https://gerrit.wikimedia.org/r/1339986 (https://phabricator.wikimedia.org/T437635) (owner: 10Dzahn) [19:18:33] (03CR) 10Dzahn: [C:03+2] create attribution.wikimedia.org for future miscweb static site [dns] - 10https://gerrit.wikimedia.org/r/1339986 (https://phabricator.wikimedia.org/T437635) (owner: 10Dzahn) [19:18:38] (03PS2) 10Dzahn: create attribution.wikimedia.org for future miscweb static site [dns] - 10https://gerrit.wikimedia.org/r/1339986 (https://phabricator.wikimedia.org/T437635) [19:20:04] (03PS2) 10JHathaway: WIP: Move DB backups from cumin1003 to cumin1004 [puppet] - 10https://gerrit.wikimedia.org/r/1341326 (https://phabricator.wikimedia.org/T427897) [19:20:11] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1341326 (https://phabricator.wikimedia.org/T427897) (owner: 10JHathaway) [19:20:49] (03PS1) 10Eevans: sessionstore1005: partition tables lost; full reimage needed [puppet] - 10https://gerrit.wikimedia.org/r/1341327 (https://phabricator.wikimedia.org/T437915) [19:22:15] !log musikanimal@deploy1003 musikanimal: Backport for [[gerrit:1341273|[CodeMirror] enable for new users (enwiki), new VE integration (global) (T288161 T432558)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [19:22:19] (03CR) 10CI reject: [V:04-1] WIP: Move DB backups from cumin1003 to cumin1004 [puppet] - 10https://gerrit.wikimedia.org/r/1341326 (https://phabricator.wikimedia.org/T427897) (owner: 10JHathaway) [19:22:21] T288161: Enable CodeMirror's Syntax Highlighting on English Wikipedia by default - https://phabricator.wikimedia.org/T288161 [19:22:21] T432558: The 2017 wikitext editor performs very poorly when CM6 is enabled - https://phabricator.wikimedia.org/T432558 [19:24:39] (03PS15) 10BCornwall: varnish: Remove reload_vcl_opts function [puppet] - 10https://gerrit.wikimedia.org/r/1298885 [19:25:08] (03CR) 10Jclark-ctr: [C:03+2] sessionstore1005: partition tables lost; full reimage needed [puppet] - 10https://gerrit.wikimedia.org/r/1341327 (https://phabricator.wikimedia.org/T437915) (owner: 10Eevans) [19:25:40] (03CR) 10BCornwall: [V:03+1] "PCC SUCCESS (DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9418/console" [puppet] - 10https://gerrit.wikimedia.org/r/1298885 (owner: 10BCornwall) [19:26:02] !log musikanimal@deploy1003 musikanimal: Continuing with deployment [19:31:59] (03PS1) 10Bking: Kerberos: promote `krb1004` to Kerberos primary [puppet] - 10https://gerrit.wikimedia.org/r/1341330 (https://phabricator.wikimedia.org/T437937) [19:32:16] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1341330 (https://phabricator.wikimedia.org/T437937) (owner: 10Bking) [19:33:34] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341331 [19:34:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [19:34:32] 10SRE-swift-storage, 10Thumbor: Gradually drop all thumbnails as a one-off clean up - https://phabricator.wikimedia.org/T379942#12319644 (10Ladsgroup) [19:34:45] 06SRE, 10Wikimedia-Mailing-lists: New WikiLatinos Mailing List - https://phabricator.wikimedia.org/T437742#12319645 (10Oscar) holy wikibureaucracy... well, I will see if a third-party option can accommodate us. [19:35:38] (03CR) 10Ladsgroup: "recheck" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1339064 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup-claude) [19:37:12] (03CR) 10Dzahn: [V:03+1] "recheck" [dns] - 10https://gerrit.wikimedia.org/r/1339986 (https://phabricator.wikimedia.org/T437635) (owner: 10Dzahn) [19:38:25] !log musikanimal@deploy1003 Finished scap sync-world: Backport for [[gerrit:1341273|[CodeMirror] enable for new users (enwiki), new VE integration (global) (T288161 T432558)]] (duration: 33m 51s) [19:38:30] T288161: Enable CodeMirror's Syntax Highlighting on English Wikipedia by default - https://phabricator.wikimedia.org/T288161 [19:38:31] T432558: The 2017 wikitext editor performs very poorly when CM6 is enabled - https://phabricator.wikimedia.org/T432558 [19:39:46] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12319664 (10Dzahn) [19:40:37] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12319666 (10Dzahn) [19:41:03] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12319673 (10Dzahn) [19:43:46] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12319684 (10Dzahn) @EAlbizzati-WMF All looking good so far:) One more thing we will need from you is a SSH key. Here are some hints how to create it: https://wikitech.wikimedia.org/wiki... [19:44:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [19:44:42] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12319689 (10Dzahn) a:03thcipriani temp assigned to Tyler for deployment access approval request [19:47:36] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment and analytics-privatedata-users for Cklimas - https://phabricator.wikimedia.org/T437653#12319695 (10Seddon) >>! In T437653#12313210, @ssingh wrote: > @Cklimas: Hi. Question about the `analytics-privatedata-users`: it is safe to... [19:47:42] 06SRE, 10Wikimedia-Mailing-lists: New WikiLatinos Mailing List - https://phabricator.wikimedia.org/T437742#12319697 (10Dzahn) Please reconsider doing this as it will create more bureaucracy when the 3rd party has different privacy policies and/or you want to import back into our own infrastructure. [19:49:38] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:50:25] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment and analytics-privatedata-users for Cklimas - https://phabricator.wikimedia.org/T437653#12319709 (10Dzahn) Tagging Data Engineering per https://wikitech.wikimedia.org/wiki/SRE/Clinic_Duty/Access_requests#analytics-privatedata-u... [19:51:10] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 13Patch-For-Review: Requesting access to deployment and analytics-privatedata-users for Cklimas - https://phabricator.wikimedia.org/T437653#12319717 (10Dzahn) [19:54:33] !log eevans@cumin1004 START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bookworm [19:55:18] !log dzahn@dns1005 START - running authdns-update [19:55:19] !log eevans@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sessionstore1005.eqiad.wmnet with OS bookworm [19:57:37] !log dzahn@dns1005 END - running authdns-update [19:58:15] !log vriley@cumin1003 START - Cookbook sre.dns.netbox [19:59:10] !log eevans@cumin1004 START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bookworm [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: #bothumor I � Unicode. All rise for UTC late backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260914T2000). [20:00:05] AGhirelli, James_F, lwatson, and aude: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:17] (03PS1) 10Dzahn: add attribution.wikimedia.org to miscweb tlsExtraSANs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341334 (https://phabricator.wikimedia.org/T437635) [20:00:45] hi [20:00:53] i can deploy mine after everyone else is done [20:02:04] !log vriley@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [ms-be1098] - vriley@cumin1003" [20:02:08] !log vriley@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [ms-be1098] - vriley@cumin1003" [20:02:08] !log vriley@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [20:02:32] AGhirelli: Did you want to self-deploy? Or I can just do it for you. [20:02:32] !log vriley@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host ms-be1098 [20:03:04] !log vriley@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1098 [20:03:41] No lwatson? [20:04:11] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jforrester@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339811 (https://phabricator.wikimedia.org/T432798) (owner: 10Aghirelli) [20:04:12] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jforrester@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338274 (https://phabricator.wikimedia.org/T432849) (owner: 10Jforrester) [20:04:25] !log vriley@cumin1003 START - Cookbook sre.hosts.provision for host ms-be1098.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [20:04:26] (03PS1) 10Dzahn: add attribution static site to helmfile, create values file [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341335 (https://phabricator.wikimedia.org/T437635) [20:05:36] (03PS3) 10Bking: sre.hadoop.roll-restart-workers: optionally skip hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1339776 (https://phabricator.wikimedia.org/T437713) [20:05:48] (03Merged) 10jenkins-bot: wmf-config: Register content/v2-beta REST module as disabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339811 (https://phabricator.wikimedia.org/T432798) (owner: 10Aghirelli) [20:06:37] (03Merged) 10jenkins-bot: wikifunctions: Move abstract fragments to mainstash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338274 (https://phabricator.wikimedia.org/T432849) (owner: 10Jforrester) [20:06:51] !log jforrester@deploy1003 Started scap sync-world: Backport for [[gerrit:1339811|wmf-config: Register content/v2-beta REST module as disabled (T432798)]], [[gerrit:1338274|wikifunctions: Move abstract fragments to mainstash (T432849)]] [20:06:58] T432798: REST API vNext: Wire standard error shape into REST framework as the default, and apply it to one vNext module - https://phabricator.wikimedia.org/T432798 [20:06:59] T432849: [Milestone 1] Observability - Build tools, document and observe behavior and evolution of our multi-layer caching system - https://phabricator.wikimedia.org/T432849 [20:07:29] (03PS2) 10Ladsgroup: [WIP] Set JPGs to be thumbnailed via VIPS if md5 starts with a [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1339064 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup-claude) [20:07:29] (03PS1) 10Ladsgroup: [DNM] Run VIPS on jpg unconditionally [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341337 (https://phabricator.wikimedia.org/T220171) [20:08:07] !log vriley@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1098.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [20:09:15] !log vriley@cumin1003 START - Cookbook sre.hosts.reimage for host ms-be1098.eqiad.wmnet with OS bullseye [20:09:31] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be1098, ms-be1099, ms-be1100 - https://phabricator.wikimedia.org/T424895#12319875 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host ms-be1098.eqiad.wmnet with... [20:12:01] (03CR) 10Bking: sre.hadoop.roll-restart-workers: optionally skip hosts (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1339776 (https://phabricator.wikimedia.org/T437713) (owner: 10Bking) [20:13:53] (03CR) 10CI reject: [V:04-1] [DNM] Run VIPS on jpg unconditionally [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341337 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup) [20:16:23] * James_F sighs at the build taking so long. [20:16:28] (03PS1) 10Dzahn: traffserver/cache: add attribution.wikimedia.org map and default caching [puppet] - 10https://gerrit.wikimedia.org/r/1341338 (https://phabricator.wikimedia.org/T437635) [20:17:44] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be1098, ms-be1099, ms-be1100 - https://phabricator.wikimedia.org/T424895#12319895 (10VRiley-WMF) [20:17:45] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 13Patch-For-Review: Requesting access to analytics-privatedata-users level 3 for derenrich - https://phabricator.wikimedia.org/T437668#12319894 (10Dzahn) [20:18:08] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 13Patch-For-Review: Requesting access to analytics-privatedata-users level 3 for derenrich - https://phabricator.wikimedia.org/T437668#12319897 (10Dzahn) Tagging Data Engineering per https://wikitech.wikimedia.org/wiki/SRE/Clinic_Duty/Access_requests#analyt... [20:19:31] (03PS1) 10Dzahn: site: add zuul1005 as a zuul executor [puppet] - 10https://gerrit.wikimedia.org/r/1319547 (https://phabricator.wikimedia.org/T427353) [20:19:32] (03CR) 10Dzahn: "Dan, does this make any sense when zuul1005 is a physical machine?" [puppet] - 10https://gerrit.wikimedia.org/r/1319547 (https://phabricator.wikimedia.org/T427353) (owner: 10Dzahn) [20:23:00] i'm ready for mine, but no hurry [20:24:18] !log jforrester@deploy1003 jforrester, aghirelli: Backport for [[gerrit:1339811|wmf-config: Register content/v2-beta REST module as disabled (T432798)]], [[gerrit:1338274|wikifunctions: Move abstract fragments to mainstash (T432849)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:24:23] T432798: REST API vNext: Wire standard error shape into REST framework as the default, and apply it to one vNext module - https://phabricator.wikimedia.org/T432798 [20:24:24] T432849: [Milestone 1] Observability - Build tools, document and observe behavior and evolution of our multi-layer caching system - https://phabricator.wikimedia.org/T432849 [20:25:20] !log jforrester@deploy1003 jforrester, aghirelli: Continuing with deployment [20:26:01] (03CR) 10BCornwall: dnsrecursor: Use a proper data type for forward zone data (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1250626 (owner: 10Majavah) [20:27:16] !log ssastry@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [20:27:51] !log ssastry@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [20:27:53] !log ssastry@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [20:28:30] !log ssastry@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [20:30:19] OK, I’m done; skipping lwatson for now as she’s not here. Over to aude. [20:30:30] (When the rollout finishes in a few seconds’ time.) [20:32:07] thanks! [20:32:12] !log jforrester@deploy1003 Finished scap sync-world: Backport for [[gerrit:1339811|wmf-config: Register content/v2-beta REST module as disabled (T432798)]], [[gerrit:1338274|wikifunctions: Move abstract fragments to mainstash (T432849)]] (duration: 25m 21s) [20:32:15] it's sure slow today [20:32:17] T432798: REST API vNext: Wire standard error shape into REST framework as the default, and apply it to one vNext module - https://phabricator.wikimedia.org/T432798 [20:32:18] T432849: [Milestone 1] Observability - Build tools, document and observe behavior and evolution of our multi-layer caching system - https://phabricator.wikimedia.org/T432849 [20:32:59] (03CR) 10TrainBranchBot: [C:03+2] "Approved by aude@deploy1003 using scap backport" [extensions/ReadingLists] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1341278 (https://phabricator.wikimedia.org/T435864) (owner: 10Jdlrobson) [20:33:36] aude: Note that all builds are doubled (as we have both PHP 8.3 and 8.5 images), which accounts for a bunch of it. [20:34:01] ok makes sense [20:34:38] (03Merged) 10jenkins-bot: [A11y] Add list semantics to ReadingList page [extensions/ReadingLists] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1341278 (https://phabricator.wikimedia.org/T435864) (owner: 10Jdlrobson) [20:34:56] !log aude@deploy1003 Started scap sync-world: Backport for [[gerrit:1341278|[A11y] Add list semantics to ReadingList page (T435864 T434923)]] [20:35:01] T435864: [Reading List - A11y Bug] Reading List and Collection lists are not semantically meaningful or accessible - https://phabricator.wikimedia.org/T435864 [20:35:02] T434923: Enable ReadingLists for all logged-in users [Sept 14] on English Wikipedia (enwiki) - https://phabricator.wikimedia.org/T434923 [20:35:38] this is the ally patch (config patch will follow) [20:39:11] !log aude@deploy1003 aude, jdlrobson: Backport for [[gerrit:1341278|[A11y] Add list semantics to ReadingList page (T435864 T434923)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:39:17] (03PS2) 10Ssingh: admin: add access for cklimas (deployment, analytics-private-data; krb) [puppet] - 10https://gerrit.wikimedia.org/r/1339879 (https://phabricator.wikimedia.org/T437653) [20:41:03] (03CR) 10Dzahn: [C:03+1] "lgtm, pending Tyler approval" [puppet] - 10https://gerrit.wikimedia.org/r/1339879 (https://phabricator.wikimedia.org/T437653) (owner: 10Ssingh) [20:41:38] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be1098, ms-be1099, ms-be1100 - https://phabricator.wikimedia.org/T424895#12320010 (10VRiley-WMF) @MatthewVernon I wanted to check on something with these servers. It seems as though the cards are 25 gig and it... [20:43:19] !log aude@deploy1003 aude, jdlrobson: Continuing with deployment [20:47:40] 10ops-eqiad, 06SRE, 10Cassandra, 06DC-Ops: sessionstore1005.eqiad.wmnet is down (again) - https://phabricator.wikimedia.org/T437915#12320016 (10RobH) [20:47:45] !log aude@deploy1003 Finished scap sync-world: Backport for [[gerrit:1341278|[A11y] Add list semantics to ReadingList page (T435864 T434923)]] (duration: 12m 49s) [20:47:50] T435864: [Reading List - A11y Bug] Reading List and Collection lists are not semantically meaningful or accessible - https://phabricator.wikimedia.org/T435864 [20:47:50] T434923: Enable ReadingLists for all logged-in users [Sept 14] on English Wikipedia (enwiki) - https://phabricator.wikimedia.org/T434923 [20:51:44] (03CR) 10TrainBranchBot: [C:03+2] "Approved by aude@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338999 (https://phabricator.wikimedia.org/T434923) (owner: 10Jdlrobson) [20:51:45] (03CR) 10TrainBranchBot: [C:03+2] "Approved by aude@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340004 (https://phabricator.wikimedia.org/T437665) (owner: 10Jdlrobson) [20:52:14] eevans@cumin1004 reimage (PID 2434482) is awaiting input [20:53:25] (03Merged) 10jenkins-bot: Enable ReadingLists for all logged-in users on English Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1338999 (https://phabricator.wikimedia.org/T434923) (owner: 10Jdlrobson) [20:54:20] (03Merged) 10jenkins-bot: Enable Reading Recommendations experiment on test wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340004 (https://phabricator.wikimedia.org/T437665) (owner: 10Jdlrobson) [20:54:34] !log aude@deploy1003 Started scap sync-world: Backport for [[gerrit:1338999|Enable ReadingLists for all logged-in users on English Wikipedia (T434923)]], [[gerrit:1340004|Enable Reading Recommendations experiment on test wiki (T437665)]] [20:54:39] T434923: Enable ReadingLists for all logged-in users [Sept 14] on English Wikipedia (enwiki) - https://phabricator.wikimedia.org/T434923 [20:54:39] T437665: Enable experiment on beta and test wiki - https://phabricator.wikimedia.org/T437665 [20:55:38] (03PS1) 10Southparkfan: rsyslog: send php8.5-fpm logs to Logstash [puppet] - 10https://gerrit.wikimedia.org/r/1341351 (https://phabricator.wikimedia.org/T435393) [20:55:47] (03CR) 10Ryan Kemper: [C:03+1] sre.hadoop.roll-restart-workers: optionally skip hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1339776 (https://phabricator.wikimedia.org/T437713) (owner: 10Bking) [20:58:34] !log aude@deploy1003 aude, jdlrobson: Backport for [[gerrit:1338999|Enable ReadingLists for all logged-in users on English Wikipedia (T434923)]], [[gerrit:1340004|Enable Reading Recommendations experiment on test wiki (T437665)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:00:05] alexsanford, Reedy, sbassett, Maryum, and manfredi: Your horoscope predicts another Weekly Security deployment window deploy. May Zuul be (nice) with you. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260914T2100). [21:00:56] !log aude@deploy1003 aude, jdlrobson: Continuing with deployment [21:02:55] Hey all - have a couple of sec patches I’d like to get out soon... [21:05:35] (03CR) 10Dzahn: [C:03+2] zuul: create location for backups of encryption keys [puppet] - 10https://gerrit.wikimedia.org/r/1339975 (https://phabricator.wikimedia.org/T436131) (owner: 10Dzahn) [21:05:37] !log aude@deploy1003 Finished scap sync-world: Backport for [[gerrit:1338999|Enable ReadingLists for all logged-in users on English Wikipedia (T434923)]], [[gerrit:1340004|Enable Reading Recommendations experiment on test wiki (T437665)]] (duration: 11m 03s) [21:05:38] (03CR) 10RLazarus: mcrouter-wancache: Shift traffic from mc-wf1002 to mc-wf1001 [puppet] - 10https://gerrit.wikimedia.org/r/1339855 (https://phabricator.wikimedia.org/T421711) (owner: 10RLazarus) [21:05:42] T434923: Enable ReadingLists for all logged-in users [Sept 14] on English Wikipedia (enwiki) - https://phabricator.wikimedia.org/T434923 [21:05:43] T437665: Enable experiment on beta and test wiki - https://phabricator.wikimedia.org/T437665 [21:11:14] (03PS2) 10Southparkfan: rsyslog: send php8.5-fpm logs to Logstash [puppet] - 10https://gerrit.wikimedia.org/r/1341351 (https://phabricator.wikimedia.org/T435393) [21:12:50] (03CR) 10Dzahn: [C:03+1] rsyslog: send php8.5-fpm logs to Logstash [puppet] - 10https://gerrit.wikimedia.org/r/1341351 (https://phabricator.wikimedia.org/T435393) (owner: 10Southparkfan) [21:26:31] !log eevans@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sessionstore1005.eqiad.wmnet with OS bookworm [21:26:45] !log eevans@cumin1004 START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bookworm [21:29:02] !log Deployed security fix for T434372 [21:29:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:29:26] !log vriley@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1098.eqiad.wmnet with OS bullseye [21:29:34] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q4:rack/setup/install ms-be1098, ms-be1099, ms-be1100 - https://phabricator.wikimedia.org/T424895#12320167 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host ms-be1098.eqiad.wmnet with OS... [21:30:18] (03CR) 10JHathaway: [C:03+2] rake_modules: support Debian 13 (Trixie) facts [puppet] - 10https://gerrit.wikimedia.org/r/1329284 (https://phabricator.wikimedia.org/T435917) (owner: 10Hashar) [21:31:50] (03CR) 10JHathaway: [C:03+2] rake_modules: stop rspec testing on Debian Buster (10) [puppet] - 10https://gerrit.wikimedia.org/r/1329250 (https://phabricator.wikimedia.org/T435917) (owner: 10Hashar) [21:33:39] 10ops-eqiad, 06DC-Ops: Moving decomm servers - https://phabricator.wikimedia.org/T437957 (10VRiley-WMF) 03NEW [21:34:09] 10ops-eqiad, 06DC-Ops: Moving decomm servers - https://phabricator.wikimedia.org/T437957#12320199 (10VRiley-WMF) 05Open→03Resolved Moved 25 decommed servers from the main cage to the storage room. [21:39:34] (03CR) 10Btullis: [C:04-1] "The change is fine in pinciple, but I think that you just need to rebase it because there are a few extra changes included by mistake." [puppet] - 10https://gerrit.wikimedia.org/r/1341330 (https://phabricator.wikimedia.org/T437937) (owner: 10Bking) [21:40:06] (03PS2) 10Bking: Kerberos: promote `krb1004` to Kerberos primary [puppet] - 10https://gerrit.wikimedia.org/r/1341330 (https://phabricator.wikimedia.org/T437937) [21:43:02] !log Deployed security fix for T435623 [21:43:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:43:05] (03CR) 10Bking: [C:03+2] sre.hadoop.roll-restart-workers: optionally skip hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1339776 (https://phabricator.wikimedia.org/T437713) (owner: 10Bking) [21:45:56] FIRING: TransportLinksInUsageNoRedundancy: eqsin inbound transports link usage is at redundancy capacity - https://wikitech.wikimedia.org/wiki/Network_monitoring#TransportLinksInUsageNoRedundancy - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/bandwidth-site-totals?var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DTransportLinksInUsageNoRedundancy [21:48:52] eevans@cumin1004 reimage (PID 2523558) is awaiting input [22:04:34] (03PS3) 10Bking: Kerberos: promote `krb1004` to Kerberos primary [puppet] - 10https://gerrit.wikimedia.org/r/1341330 (https://phabricator.wikimedia.org/T437937) [22:05:36] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1341330 (https://phabricator.wikimedia.org/T437937) (owner: 10Bking) [22:11:26] (03PS4) 10Bking: Kerberos: promote `krb1004` to Kerberos primary [puppet] - 10https://gerrit.wikimedia.org/r/1341330 (https://phabricator.wikimedia.org/T437937) [22:13:44] (03PS2) 10Ladsgroup: [DNM] Run VIPS on jpg unconditionally [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341337 (https://phabricator.wikimedia.org/T220171) [22:18:25] (03PS5) 10Bking: Kerberos: promote `krb1004` to Kerberos primary [puppet] - 10https://gerrit.wikimedia.org/r/1341330 (https://phabricator.wikimedia.org/T437937) [22:19:16] (03CR) 10Bking: "Sorry for the trouble, I think I've removed all the unrelated changes." [puppet] - 10https://gerrit.wikimedia.org/r/1341330 (https://phabricator.wikimedia.org/T437937) (owner: 10Bking) [22:20:20] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1341330 (https://phabricator.wikimedia.org/T437937) (owner: 10Bking) [22:22:16] !log eevans@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sessionstore1005.eqiad.wmnet with OS bookworm [22:22:33] !log eevans@cumin1004 START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bookworm [22:23:23] Anything going on that'd get in the way of me doing a really quick back port? [22:23:38] (03CR) 10CI reject: [V:04-1] [DNM] Run VIPS on jpg unconditionally [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341337 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup) [22:23:43] (03PS1) 10DLynch: VisualEditor: don't register settings tool in wikitextCommandRegistry [extensions/CodeMirror] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1341385 (https://phabricator.wikimedia.org/T437810) [22:26:07] It seems not, so... [22:26:44] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kemayo@deploy1003 using scap backport" [extensions/CodeMirror] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1341385 (https://phabricator.wikimedia.org/T437810) (owner: 10DLynch) [22:31:27] (03Merged) 10jenkins-bot: VisualEditor: don't register settings tool in wikitextCommandRegistry [extensions/CodeMirror] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1341385 (https://phabricator.wikimedia.org/T437810) (owner: 10DLynch) [22:31:37] (03PS3) 10Ladsgroup: [DNM] Run VIPS on jpg unconditionally [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341337 (https://phabricator.wikimedia.org/T220171) [22:31:43] !log kemayo@deploy1003 Started scap sync-world: Backport for [[gerrit:1341385|VisualEditor: don't register settings tool in wikitextCommandRegistry (T437810)]] [22:31:47] T437810: Source editor on mobile gets stuck: TypeError: can't access property "register", ve.ui.wikitextCommandRegistry is undefined - https://phabricator.wikimedia.org/T437810 [22:36:08] !log kemayo@deploy1003 kemayo: Backport for [[gerrit:1341385|VisualEditor: don't register settings tool in wikitextCommandRegistry (T437810)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:36:35] !log kemayo@deploy1003 kemayo: Continuing with deployment [22:38:58] (03CR) 10CI reject: [V:04-1] [DNM] Run VIPS on jpg unconditionally [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341337 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup) [22:39:59] (03PS4) 10Ladsgroup: [DNM] Run VIPS on jpg unconditionally [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341337 (https://phabricator.wikimedia.org/T220171) [22:41:06] !log kemayo@deploy1003 Finished scap sync-world: Backport for [[gerrit:1341385|VisualEditor: don't register settings tool in wikitextCommandRegistry (T437810)]] (duration: 09m 22s) [22:41:09] T437810: Source editor on mobile gets stuck: TypeError: can't access property "register", ve.ui.wikitextCommandRegistry is undefined - https://phabricator.wikimedia.org/T437810 [22:41:28] Okay, I'm all done. [22:48:55] (03CR) 10CI reject: [V:04-1] [DNM] Run VIPS on jpg unconditionally [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341337 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup) [22:53:11] (03PS1) 10Ladsgroup-claude: Introduce concept of thumbnails with expiry for recently uploaded files [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341399 (https://phabricator.wikimedia.org/T433964) [22:56:26] (03CR) 10Btullis: [C:03+1] "Looks good, thanks." [puppet] - 10https://gerrit.wikimedia.org/r/1341330 (https://phabricator.wikimedia.org/T437937) (owner: 10Bking) [22:56:59] (03PS5) 10Ladsgroup: [DNM] Run VIPS on jpg unconditionally [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341337 (https://phabricator.wikimedia.org/T220171) [22:58:16] eevans@cumin1004 reimage (PID 2580431) is awaiting input [23:00:04] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260914T2300) [23:01:52] !log eevans@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sessionstore1005.eqiad.wmnet with OS bookworm [23:03:06] (03CR) 10Ladsgroup: "recheck" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341399 (https://phabricator.wikimedia.org/T433964) (owner: 10Ladsgroup-claude) [23:04:14] (03CR) 10CI reject: [V:04-1] [DNM] Run VIPS on jpg unconditionally [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341337 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup) [23:06:30] 10ops-eqiad, 06SRE, 10Cassandra, 06DC-Ops: sessionstore1005.eqiad.wmnet is down (again) - https://phabricator.wikimedia.org/T437915#12320406 (10Jclark-ctr) Service Request with Dell 231696528 [23:06:39] (03PS6) 10Ladsgroup: [DNM] Run VIPS on jpg unconditionally [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341337 (https://phabricator.wikimedia.org/T220171) [23:09:19] I would like to deploy a change now, if the readers window is unoccupied [23:11:11] (03CR) 10CI reject: [V:04-1] Introduce concept of thumbnails with expiry for recently uploaded files [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341399 (https://phabricator.wikimedia.org/T433964) (owner: 10Ladsgroup-claude) [23:12:28] (03CR) 10TrainBranchBot: [C:03+2] "Approved by tstarling@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324966 (https://phabricator.wikimedia.org/T421436) (owner: 10Tim Starling) [23:12:39] (03CR) 10CI reject: [V:04-1] [DNM] Run VIPS on jpg unconditionally [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341337 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup) [23:13:32] (03Merged) 10jenkins-bot: Enable Produnto on pilot wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1324966 (https://phabricator.wikimedia.org/T421436) (owner: 10Tim Starling) [23:13:47] !log tstarling@deploy1003 Started scap sync-world: Backport for [[gerrit:1324966|Enable Produnto on pilot wikis (T421436)]] [23:13:50] T421436: Deploy Produnto extension to production - https://phabricator.wikimedia.org/T421436 [23:16:40] FIRING: SystemdUnitFailed: production-images-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:17:47] !log tstarling@deploy1003 tstarling: Backport for [[gerrit:1324966|Enable Produnto on pilot wikis (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [23:20:56] RESOLVED: TransportLinksInUsageNoRedundancy: eqsin inbound transports link usage is at redundancy capacity - https://wikitech.wikimedia.org/wiki/Network_monitoring#TransportLinksInUsageNoRedundancy - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/bandwidth-site-totals?var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DTransportLinksInUsageNoRedundancy [23:22:17] !log tstarling@deploy1003 tstarling: Continuing with deployment [23:26:46] !log tstarling@deploy1003 Finished scap sync-world: Backport for [[gerrit:1324966|Enable Produnto on pilot wikis (T421436)]] (duration: 12m 59s) [23:26:50] T421436: Deploy Produnto extension to production - https://phabricator.wikimedia.org/T421436 [23:28:33] (03PS7) 10Ladsgroup: [DNM] Run VIPS on jpg unconditionally [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341337 (https://phabricator.wikimedia.org/T220171) [23:29:29] (03CR) 10CI reject: [V:04-1] [DNM] Run VIPS on jpg unconditionally [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341337 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup) [23:35:44] 06SRE, 10Beta-Cluster-Infrastructure, 06Traffic: Name new CDN servers deployment-cp-(text|upload)0x instead of deployment-cache-(text|upload)0x - https://phabricator.wikimedia.org/T280393#12320486 (10BCornwall) 05Open→03Resolved a:03BCornwall deployment-cp-text09 and deployment-cp-upload09 are up a... [23:40:58] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1341409 [23:40:58] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1341409 (owner: 10TrainBranchBot) [23:41:38] !log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bookworm [23:41:52] 10ops-eqiad, 06SRE, 10Cassandra, 06DC-Ops: sessionstore1005.eqiad.wmnet is down (again) - https://phabricator.wikimedia.org/T437915#12320495 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host sessionstore1005.eqiad.wmnet with OS bookworm [23:42:19] (03PS2) 10Ladsgroup-claude: Introduce concept of thumbnails with expiry for recently uploaded files [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341399 (https://phabricator.wikimedia.org/T433964) [23:43:07] (03CR) 10Ladsgroup: "recheck" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341399 (https://phabricator.wikimedia.org/T433964) (owner: 10Ladsgroup-claude) [23:44:06] (03PS8) 10Ladsgroup: [DNM] Run VIPS on jpg unconditionally [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341337 (https://phabricator.wikimedia.org/T220171) [23:46:09] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 13Patch-For-Review: Requesting access to deployment and analytics-privatedata-users for Cklimas - https://phabricator.wikimedia.org/T437653#12320516 (10thcipriani) Reason for access makes sense for `deployment` group membership. Approved! --- @Cklimas som... [23:47:07] (03CR) 10CI reject: [V:04-1] [DNM] Run VIPS on jpg unconditionally [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1341337 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup) [23:47:09] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12320518 (10thcipriani) Reason for access makes sense for `deployment` group membership. Approved! --- @EAlbizzati-WMF some additional info for you: - If you ever need to deploy Media... [23:48:03] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment and restricted for WRai-WMF - https://phabricator.wikimedia.org/T437652#12320519 (10thcipriani) Reason for access makes sense for `deployment` group membership. Approved! --- @WRai-WMF some additional info for you: - If you... [23:49:36] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1341409 (owner: 10TrainBranchBot) [23:49:38] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [23:51:56] FIRING: TransportLinksInUsageNoRedundancy: eqsin inbound transports link usage is at redundancy capacity - https://wikitech.wikimedia.org/wiki/Network_monitoring#TransportLinksInUsageNoRedundancy - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/bandwidth-site-totals?var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DTransportLinksInUsageNoRedundancy [23:52:31] (03PS3) 10Ladsgroup: Set JPGs to be thumbnailed via VIPS if md5 starts with a [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1339064 (https://phabricator.wikimedia.org/T220171) (owner: 10Ladsgroup-claude) [23:54:00] (03CR) 10Jforrester: [C:03+1] mcrouter-wancache: Shift traffic from mc-wf1002 to mc-wf1001 [puppet] - 10https://gerrit.wikimedia.org/r/1339855 (https://phabricator.wikimedia.org/T421711) (owner: 10RLazarus) [23:55:18] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment and restricted for cooltey - https://phabricator.wikimedia.org/T437658#12320527 (10thcipriani) Reason for access makes sense for `deployment` group membership. Approved! No reason to add to `restricted` as `deployment` can do... [23:55:31] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to releasers-mobile for LPetty-WMF - https://phabricator.wikimedia.org/T437662#12320528 (10thcipriani) >>! In T437662#12312796, @Dzahn wrote: > Hello @thcipriani or @dancy > > this will require your approval as owners of the `releasers-mob... [23:59:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed