[00:24:57] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [00:40:16] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 627.36 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [01:05:58] RECOVERY - MariaDB Replica Lag: s4 on db1265 is OK: OK slave_sql_lag Replication lag: 0.17 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [01:08:47] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1345350 [01:08:47] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1345350 (owner: 10TrainBranchBot) [01:17:05] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for kafka expires in 29d 11h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [01:17:53] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1345350 (owner: 10TrainBranchBot) [01:20:06] (03PS1) 10Andrew Bogott: wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 [01:23:51] (03CR) 10CI reject: [V:04-1] wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 (owner: 10Andrew Bogott) [01:25:14] (03PS2) 10Andrew Bogott: wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 [01:27:23] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [01:28:11] (03CR) 10CI reject: [V:04-1] wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 (owner: 10Andrew Bogott) [01:31:48] (03PS3) 10Andrew Bogott: wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 [01:34:28] (03CR) 10CI reject: [V:04-1] wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 (owner: 10Andrew Bogott) [01:41:48] (03PS4) 10Andrew Bogott: wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 [01:44:21] (03CR) 10CI reject: [V:04-1] wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 (owner: 10Andrew Bogott) [01:47:49] (03PS5) 10Andrew Bogott: wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 [01:50:28] (03CR) 10CI reject: [V:04-1] wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 (owner: 10Andrew Bogott) [01:59:52] (03PS6) 10Andrew Bogott: wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 [02:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260927T0700) [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260928T0200) [02:00:21] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:01:58] PROBLEM - MariaDB Replica Lag: s3 on db1265 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 646.09 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:02:24] (03CR) 10CI reject: [V:04-1] wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 (owner: 10Andrew Bogott) [02:08:10] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 48s) [02:09:03] (03PS7) 10Andrew Bogott: wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 [02:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:11:53] (03CR) 10CI reject: [V:04-1] wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 (owner: 10Andrew Bogott) [02:13:28] FIRING: [2x] ProbeDown: Service kafka-logging1003:9093 has failed probes (tcp_kafka_broker_tls_kafka_logging1003_ip4) - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [02:16:27] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:18:17] (03PS8) 10Andrew Bogott: wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 [02:19:16] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.16 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:20:59] (03CR) 10CI reject: [V:04-1] wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 (owner: 10Andrew Bogott) [02:24:08] (03PS9) 10Andrew Bogott: wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 [02:26:44] (03CR) 10CI reject: [V:04-1] wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 (owner: 10Andrew Bogott) [02:39:08] (03PS10) 10Andrew Bogott: wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 [02:41:41] (03CR) 10CI reject: [V:04-1] wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 (owner: 10Andrew Bogott) [02:47:54] (03PS11) 10Andrew Bogott: wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 [02:50:33] (03CR) 10CI reject: [V:04-1] wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 (owner: 10Andrew Bogott) [02:53:58] (03PS12) 10Andrew Bogott: wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 [02:56:33] (03CR) 10CI reject: [V:04-1] wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 (owner: 10Andrew Bogott) [02:59:53] (03PS13) 10Andrew Bogott: wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 [03:21:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:39:38] (03PS2) 10Milazg: Add URLs for MobileAppRedirect Special Page [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339561 (https://phabricator.wikimedia.org/T434930) [03:41:58] RECOVERY - MariaDB Replica Lag: s3 on db1265 is OK: OK slave_sql_lag Replication lag: 0.13 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [04:24:57] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [04:53:40] FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker1053:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1053 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [05:03:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker1053:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1053 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [05:17:05] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for kafka expires in 29d 7h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [05:27:23] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [05:59:35] (03CR) 10Awight: "Task ID? It would be great to add some explanation of what was squashed" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345339 (owner: 10Matthias Mullie) [06:02:54] (03CR) 10Awight: "Also, this adds a new message—is that safe to do as a backport? I think I've seen l10n-cache logs when deploying using spiderpig, in whic" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345339 (owner: 10Matthias Mullie) [06:09:43] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply [06:09:46] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply [06:10:05] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook-next: apply [06:10:08] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook-next: apply [06:13:28] FIRING: [2x] ProbeDown: Service kafka-logging1003:9093 has failed probes (tcp_kafka_broker_tls_kafka_logging1003_ip4) - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [06:16:27] FIRING: [2x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:19:07] (03PS2) 10Ayounsi: rack depool hiera, add policy: manual and remaining server_pool keys [puppet] - 10https://gerrit.wikimedia.org/r/1344205 (https://phabricator.wikimedia.org/T327300) [06:20:16] (03CR) 10Ayounsi: rack depool hiera, add policy: manual and remaining server_pool keys (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1344205 (https://phabricator.wikimedia.org/T327300) (owner: 10Ayounsi) [06:24:02] (03PS1) 10Brouberol: site: remove ganeti-jumbo1* host [puppet] - 10https://gerrit.wikimedia.org/r/1345357 (https://phabricator.wikimedia.org/T439241) [06:24:04] (03PS1) 10Brouberol: site: set dse-k8s-worker200[6-8] as insetup hosts [puppet] - 10https://gerrit.wikimedia.org/r/1345358 (https://phabricator.wikimedia.org/T439241) [06:24:07] (03PS1) 10Brouberol: site/dse-k8s-worker2006: assign dse_k8s::worker role [puppet] - 10https://gerrit.wikimedia.org/r/1345359 (https://phabricator.wikimedia.org/T439241) [06:24:09] (03PS1) 10Brouberol: site/dse-k8s-worker2007: assign dse_k8s::worker role [puppet] - 10https://gerrit.wikimedia.org/r/1345360 (https://phabricator.wikimedia.org/T439241) [06:24:11] (03PS1) 10Brouberol: site/dse-k8s-worker2008: assign dse_k8s::worker role [puppet] - 10https://gerrit.wikimedia.org/r/1345361 (https://phabricator.wikimedia.org/T439241) [06:24:13] (03PS1) 10Brouberol: site: remove ganeti-jumbo role entries [puppet] - 10https://gerrit.wikimedia.org/r/1345362 (https://phabricator.wikimedia.org/T439241) [06:27:59] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply [06:28:03] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply [06:44:47] (03PS1) 10Matthias Mullie: Remove reliance on ParserOutput [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345369 (https://phabricator.wikimedia.org/T439182) [06:44:54] (03PS1) 10Matthias Mullie: Remove reliance on file repo [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345370 (https://phabricator.wikimedia.org/T439182) [06:45:06] (03PS1) 10Matthias Mullie: Fix jump to image break in carousel retest [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345371 (https://phabricator.wikimedia.org/T439252) [06:47:14] (03CR) 10CI reject: [V:04-1] Remove reliance on file repo [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345370 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [06:47:33] (03CR) 10Matthias Mullie: "I thought l10n is rebuilt, but I'll update this anyway. The i18n messages were intentional, but we're still missing some so will need anot" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345339 (owner: 10Matthias Mullie) [06:47:36] (03Abandoned) 10Matthias Mullie: Squashed diff to master [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345339 (owner: 10Matthias Mullie) [06:52:20] (03CR) 10Brouberol: [C:03+1] "LG!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344034 (https://phabricator.wikimedia.org/T435591) (owner: 10Btullis) [06:53:12] (03PS2) 10Brouberol: ceph::osd: disable the OSD service at deletion [cookbooks] - 10https://gerrit.wikimedia.org/r/1344626 (https://phabricator.wikimedia.org/T438823) [06:53:19] (03CR) 10Brouberol: ceph::osd: disable the OSD service at deletion (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1344626 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [06:55:45] (03CR) 10Brouberol: [C:03+1] ceph: Enable STS on the codfw rados gateway [puppet] - 10https://gerrit.wikimedia.org/r/1345117 (https://phabricator.wikimedia.org/T435590) (owner: 10Btullis) [07:00:04] Amir1, urbanecm, and awight: How many deployers does it take to do UTC morning backport window deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260928T0700). [07:00:04] awight, matthiasmullie, and MichaelG_WMF: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:00:06] o/ [07:02:01] hi [07:02:26] (03CR) 10Matthias Mullie: [C:03+2] Remove reliance on ParserOutput [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345369 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [07:03:00] Looks like awight is not here yet - I'll get started in the meantime [07:03:36] (03Merged) 10jenkins-bot: Remove reliance on ParserOutput [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345369 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [07:03:45] (03PS2) 10Matthias Mullie: Remove reliance on file repo [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345370 (https://phabricator.wikimedia.org/T439182) [07:04:18] matthiasmullie: fine with me. In case awight does not appear, could you deploy my changes as well after you're done? [07:04:45] sure [07:04:49] thanks! [07:05:25] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345070 (owner: 10Matthias Mullie) [07:05:25] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345371 (https://phabricator.wikimedia.org/T439252) (owner: 10Matthias Mullie) [07:05:25] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345370 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [07:06:20] (03Merged) 10jenkins-bot: [MultimediaViewer] Testwiki use same carousel config as other wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345070 (owner: 10Matthias Mullie) [07:06:37] Late but present. I can deploy last [07:07:41] (03Merged) 10jenkins-bot: Fix jump to image break in carousel retest [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345371 (https://phabricator.wikimedia.org/T439252) (owner: 10Matthias Mullie) [07:07:43] (03Merged) 10jenkins-bot: Remove reliance on file repo [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345370 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [07:08:05] !log mlitn@deploy1003 Started scap sync-world: Backport for [[gerrit:1345070|[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371|Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369|Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370|Remove reliance on file repo (T439182)]] [07:08:10] T439252: [Image Browsing] "Scroll to image" does not work with legacy parser - https://phabricator.wikimedia.org/T439252 [07:08:11] T439182: [Regression] MediaViewer hook adds 30% to Backend Pageview Tining on mobile (300 -> 400ms) - https://phabricator.wikimedia.org/T439182 [07:16:45] (03PS1) 10Muehlenhoff: Update access metadata for jniren-ctr [puppet] - 10https://gerrit.wikimedia.org/r/1345381 [07:17:45] (03PS5) 10Elukey: Move the pki service entry to service_setup [puppet] - 10https://gerrit.wikimedia.org/r/1338937 (https://phabricator.wikimedia.org/T436809) [07:20:27] (03PS6) 10Elukey: Move the pki service entry to service_setup [puppet] - 10https://gerrit.wikimedia.org/r/1338937 (https://phabricator.wikimedia.org/T436809) [07:21:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:21:42] (03CR) 10Elukey: "Removed the "monitor" section to leave the "probes" one, lemme know if it makes sense!" [puppet] - 10https://gerrit.wikimedia.org/r/1338937 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [07:24:11] (03CR) 10Giuseppe Lavagetto: [C:03+2] Add asyncio implementation [software/python-poolcounter] - 10https://gerrit.wikimedia.org/r/980918 (https://phabricator.wikimedia.org/T338297) (owner: 10Giuseppe Lavagetto) [07:26:45] !log mlitn@deploy1003 mlitn: Backport for [[gerrit:1345070|[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371|Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369|Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370|Remove reliance on file repo (T439182)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verifi [07:26:45] ed there. [07:26:50] T439252: [Image Browsing] "Scroll to image" does not work with legacy parser - https://phabricator.wikimedia.org/T439252 [07:26:50] T439182: [Regression] MediaViewer hook adds 30% to Backend Pageview Tining on mobile (300 -> 400ms) - https://phabricator.wikimedia.org/T439182 [07:26:50] (03CR) 10Ayounsi: [C:03+2] rack depool hiera, add policy: manual and remaining server_pool keys [puppet] - 10https://gerrit.wikimedia.org/r/1344205 (https://phabricator.wikimedia.org/T327300) (owner: 10Ayounsi) [07:27:16] (I'm no longer going to deploy my config change. Removing now.) [07:27:35] !log mlitn@deploy1003 mlitn: Continuing with deployment [07:28:00] (03CR) 10CI reject: [V:04-1] Add asyncio implementation [software/python-poolcounter] - 10https://gerrit.wikimedia.org/r/980918 (https://phabricator.wikimedia.org/T338297) (owner: 10Giuseppe Lavagetto) [07:35:25] (03CR) 10Muehlenhoff: [C:03+2] Update access metadata for jniren-ctr [puppet] - 10https://gerrit.wikimedia.org/r/1345381 (owner: 10Muehlenhoff) [07:38:04] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_codfw and not P{cp2059.*} and A:cp - 3.2.23 upgrade (T438828) [07:38:07] T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828 [07:38:17] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 3.2.23 upgrade (T438828) [07:40:12] !log mlitn@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345070|[MultimediaViewer] Testwiki use same carousel config as other wikis]], [[gerrit:1345371|Fix jump to image break in carousel retest (T439252)]], [[gerrit:1345369|Remove reliance on ParserOutput (T439182)]], [[gerrit:1345370|Remove reliance on file repo (T439182)]] (duration: 32m 06s) [07:40:16] T439252: [Image Browsing] "Scroll to image" does not work with legacy parser - https://phabricator.wikimedia.org/T439252 [07:40:17] T439182: [Regression] MediaViewer hook adds 30% to Backend Pageview Tining on mobile (300 -> 400ms) - https://phabricator.wikimedia.org/T439182 [07:40:19] This took forever [07:40:20] MichaelG_WMF: Ok if I combine both of your patches at once? [07:40:36] matthiasmullie: sure [07:40:41] thanks :) [07:41:00] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343611 (https://phabricator.wikimedia.org/T432126) (owner: 10Michael Große) [07:41:00] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344594 (https://phabricator.wikimedia.org/T437339) (owner: 10Michael Große) [07:42:03] (03Merged) 10jenkins-bot: Growth: Set minimum registration date for A/B test [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343611 (https://phabricator.wikimedia.org/T432126) (owner: 10Michael Große) [07:43:49] (03CR) 10Klausman: [C:03+1] dse-k8s: allow unauthenticated reads of the OIDC discovery endpoints [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344034 (https://phabricator.wikimedia.org/T435591) (owner: 10Btullis) [07:44:01] (03CR) 10Klausman: [C:03+1] site: remove ganeti-jumbo1* host [puppet] - 10https://gerrit.wikimedia.org/r/1345357 (https://phabricator.wikimedia.org/T439241) (owner: 10Brouberol) [07:44:33] (03Merged) 10jenkins-bot: EarlyOnboarding: record account setup motivation in `action_context` [extensions/WikimediaEvents] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344594 (https://phabricator.wikimedia.org/T437339) (owner: 10Michael Große) [07:44:52] !log mlitn@deploy1003 Started scap sync-world: Backport for [[gerrit:1343611|Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594|EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] [07:44:57] T432126: Experiment: DE1.3.1 Early onboarding & account set-up - https://phabricator.wikimedia.org/T432126 [07:44:57] T437339: Recommended reading exp: Instrument user interaction on module - https://phabricator.wikimedia.org/T437339 [07:45:26] (03CR) 10Klausman: [C:03+1] site: set dse-k8s-worker200[6-8] as insetup hosts [puppet] - 10https://gerrit.wikimedia.org/r/1345358 (https://phabricator.wikimedia.org/T439241) (owner: 10Brouberol) [07:45:40] (03CR) 10Klausman: [C:03+1] site/dse-k8s-worker2006: assign dse_k8s::worker role [puppet] - 10https://gerrit.wikimedia.org/r/1345359 (https://phabricator.wikimedia.org/T439241) (owner: 10Brouberol) [07:45:41] (03CR) 10Brouberol: [C:03+2] site: remove ganeti-jumbo1* host [puppet] - 10https://gerrit.wikimedia.org/r/1345357 (https://phabricator.wikimedia.org/T439241) (owner: 10Brouberol) [07:45:45] (03CR) 10Brouberol: [C:03+2] site: set dse-k8s-worker200[6-8] as insetup hosts [puppet] - 10https://gerrit.wikimedia.org/r/1345358 (https://phabricator.wikimedia.org/T439241) (owner: 10Brouberol) [07:45:56] (03CR) 10Klausman: [C:03+1] site/dse-k8s-worker2007: assign dse_k8s::worker role [puppet] - 10https://gerrit.wikimedia.org/r/1345360 (https://phabricator.wikimedia.org/T439241) (owner: 10Brouberol) [07:46:09] (03CR) 10Klausman: [C:03+1] site/dse-k8s-worker2008: assign dse_k8s::worker role [puppet] - 10https://gerrit.wikimedia.org/r/1345361 (https://phabricator.wikimedia.org/T439241) (owner: 10Brouberol) [07:46:28] (03CR) 10Klausman: [C:03+1] site: remove ganeti-jumbo role entries [puppet] - 10https://gerrit.wikimedia.org/r/1345362 (https://phabricator.wikimedia.org/T439241) (owner: 10Brouberol) [07:47:44] !log brouberol@cumin1004 START - Cookbook sre.hosts.rename from ganeti-jumbo2001 to dse-k8s-worker2006 [07:48:06] !log brouberol@cumin1004 START - Cookbook sre.dns.netbox [07:49:09] !log mlitn@deploy1003 migr, mlitn: Backport for [[gerrit:1343611|Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594|EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:49:42] 10SRE-tools, 06Infrastructure-Foundations, 13Patch-For-Review: Cookbook for rack depool - https://phabricator.wikimedia.org/T327300#12367696 (10ayounsi) 05Open→03Resolved a:03ayounsi Going to close that one, the pool and depool actions have been tested, now supports also the "manual" policy. Some r... [07:49:43] MichaelG_WMF: it's on testservers - can you confirm? [07:49:49] * MichaelG_WMF is testing [07:51:37] matthiasmullie: Looks good. Ready to move forward from my side 👍 [07:51:48] !log mlitn@deploy1003 migr, mlitn: Continuing with deployment [07:52:02] !log brouberol@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2001 to dse-k8s-worker2006 - brouberol@cumin1004" [07:52:17] !log brouberol@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2001 to dse-k8s-worker2006 - brouberol@cumin1004" [07:52:17] !log brouberol@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [07:52:17] !log brouberol@cumin1004 START - Cookbook sre.dns.wipe-cache dse-k8s-worker2006 on all recursors [07:52:20] !log brouberol@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2006 on all recursors [07:52:21] !log brouberol@cumin1004 START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2006 [07:52:34] !log brouberol@cumin1004 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2006 [07:52:58] (03PS1) 10Elukey: sre.hosts.provision: update supported vendors in docstring [cookbooks] - 10https://gerrit.wikimedia.org/r/1345483 [07:53:10] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2001 to dse-k8s-worker2006 [07:53:47] (03CR) 10Hashar: "Does `prometheus::blackbox::check::tcp` run on the `localhost` or from an external monitoring host? TCP port 4730 is the Gearman protocol" [puppet] - 10https://gerrit.wikimedia.org/r/1343955 (https://phabricator.wikimedia.org/T384939) (owner: 10Hnowlan) [07:58:05] !log brouberol@cumin1004 START - Cookbook sre.hosts.reimage for host dse-k8s-worker2006.codfw.wmnet with OS bookworm [07:59:04] !log mlitn@deploy1003 Finished scap sync-world: Backport for [[gerrit:1343611|Growth: Set minimum registration date for A/B test (T432126)]], [[gerrit:1344594|EarlyOnboarding: record account setup motivation in `action_context` (T437339)]] (duration: 14m 12s) [07:59:09] T432126: Experiment: DE1.3.1 Early onboarding & account set-up - https://phabricator.wikimedia.org/T432126 [07:59:09] T437339: Recommended reading exp: Instrument user interaction on module - https://phabricator.wikimedia.org/T437339 [07:59:16] MichaelG_WMF: Done [07:59:24] matthiasmullie: Great, thanks! [07:59:31] !log UTC morning backport window complete [07:59:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:00:38] (03CR) 10Elukey: [C:03+2] sre.hosts.provision: update supported vendors in docstring [cookbooks] - 10https://gerrit.wikimedia.org/r/1345483 (owner: 10Elukey) [08:01:42] (03CR) 10JMeybohm: [C:03+1] gVisor: disable on bookworm in containerd [puppet] - 10https://gerrit.wikimedia.org/r/1342961 (https://phabricator.wikimedia.org/T436649) (owner: 10Giuseppe Lavagetto) [08:02:12] (03PS5) 10JMeybohm: gVisor: use kvm as a platform when hardware virtualization is available [puppet] - 10https://gerrit.wikimedia.org/r/1342962 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [08:02:15] (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342962 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [08:02:23] (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342963 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [08:04:41] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 3.2.23 upgrade (T438828) [08:04:45] T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828 [08:08:12] (03CR) 10Dpogorzelski: liftwing-studio: back the database with ferretdb (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345057 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [08:08:32] (03CR) 10Dpogorzelski: [C:03+2] liftwing-studio: back the database with ferretdb [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345057 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [08:08:44] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] liftwing-studio: back the database with ferretdb [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345057 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [08:09:07] (03CR) 10Dpogorzelski: [C:03+2] liftwing-studio: replace Open WebUI with LibreChat [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345058 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [08:10:00] FIRING: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker2006 has a BGP session which is not in the 'established' state. [08:16:06] !log installing Exim security updates [08:16:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:16:45] (03PS1) 10Daniel Kertesz: Revert "hiera: enable HAProxy's persistent stats on two magru nodes" [puppet] - 10https://gerrit.wikimedia.org/r/1345486 [08:17:48] (03CR) 10CI reject: [V:04-1] Revert "hiera: enable HAProxy's persistent stats on two magru nodes" [puppet] - 10https://gerrit.wikimedia.org/r/1345486 (owner: 10Daniel Kertesz) [08:18:38] !log brouberol@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-worker2006.codfw.wmnet with OS bookworm [08:18:54] !log brouberol@cumin1004 START - Cookbook sre.hosts.reimage for host dse-k8s-worker2006.codfw.wmnet with OS bookworm [08:20:38] (03CR) 10Fabfur: [C:03+1] "+1 for me to merge when CI is happy" [puppet] - 10https://gerrit.wikimedia.org/r/1345486 (owner: 10Daniel Kertesz) [08:20:43] (03PS4) 10Giuseppe Lavagetto: gVisor: disable on bookworm in containerd [puppet] - 10https://gerrit.wikimedia.org/r/1342961 (https://phabricator.wikimedia.org/T436649) [08:20:43] (03PS6) 10Giuseppe Lavagetto: gVisor: use kvm as a platform when hardware virtualization is available [puppet] - 10https://gerrit.wikimedia.org/r/1342962 (https://phabricator.wikimedia.org/T438287) [08:20:43] (03PS4) 10Giuseppe Lavagetto: kubernetes::node: add gvisor label based on hardware virtualization [puppet] - 10https://gerrit.wikimedia.org/r/1342963 (https://phabricator.wikimedia.org/T438287) [08:20:50] (03PS2) 10Daniel Kertesz: Revert "hiera: enable HAProxy's persistent stats on two magru nodes" [puppet] - 10https://gerrit.wikimedia.org/r/1345486 [08:20:51] PROBLEM - librenms.wikimedia.org tls expiry on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [08:21:07] PROBLEM - SSH on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [08:21:11] PROBLEM - librenms.wikimedia.org requires authentication on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [08:22:37] (03CR) 10Giuseppe Lavagetto: [V:03+1] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9481/console" [puppet] - 10https://gerrit.wikimedia.org/r/1342962 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [08:22:37] (03CR) 10CI reject: [V:04-1] Revert "hiera: enable HAProxy's persistent stats on two magru nodes" [puppet] - 10https://gerrit.wikimedia.org/r/1345486 (owner: 10Daniel Kertesz) [08:24:58] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [08:25:02] (03PS3) 10Daniel Kertesz: Revert "hiera: enable HAProxy's persistent stats on two magru nodes" [puppet] - 10https://gerrit.wikimedia.org/r/1345486 (https://phabricator.wikimedia.org/T343000) [08:26:23] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_codfw and not P{cp2059.*} and A:cp - 3.2.23 upgrade (T438828) [08:26:26] T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828 [08:29:42] FIRING: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [08:29:42] !log brouberol@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2006.codfw.wmnet with reason: host reimage [08:30:04] (03CR) 10Daniel Kertesz: [C:03+2] Revert "hiera: enable HAProxy's persistent stats on two magru nodes" [puppet] - 10https://gerrit.wikimedia.org/r/1345486 (https://phabricator.wikimedia.org/T343000) (owner: 10Daniel Kertesz) [08:33:07] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2006.codfw.wmnet with reason: host reimage [08:36:53] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [08:42:20] 06SRE, 10observability, 06Traffic, 13Patch-For-Review: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12368068 (10dkertesz) Reverted the change that enabled persistent stats on cp7001 and cp7011 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1345486 There were mo... [08:44:13] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [08:44:35] (03PS1) 10Muehlenhoff: proton: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345488 [08:48:44] RECOVERY - librenms.wikimedia.org tls expiry on netmon2002 is OK: OK - Certificate librenms.wikimedia.org will expire on Mon 09 Nov 2026 02:18:41 AM GMT +0000. https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [08:50:13] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2006.codfw.wmnet with OS bookworm [08:51:15] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-liftwing-studio: sync [08:51:23] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-liftwing-studio: sync [08:51:50] PROBLEM - librenms.wikimedia.org tls expiry on netmon2002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [08:52:02] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync [08:52:04] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync [08:52:22] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [08:53:40] RECOVERY - librenms.wikimedia.org tls expiry on netmon2002 is OK: OK - Certificate librenms.wikimedia.org will expire on Mon 09 Nov 2026 02:18:41 AM GMT +0000. https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [08:54:04] RECOVERY - librenms.wikimedia.org requires authentication on netmon2002 is OK: HTTP OK: Status line output matched HTTP/1.1 302 - 701 bytes in 0.133 second response time https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [08:56:01] (03CR) 10Jelto: [C:03+2] service::catalog: Set ipip_encapsulation for k8s-ingress-wikikube in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1311444 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [08:58:44] (03CR) 10Gmodena: [C:03+2] wdqs: widen proxy readiness probe timeout/failureThreshold [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345098 (https://phabricator.wikimedia.org/T439248) (owner: 10Gmodena) [09:00:20] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [09:00:31] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [09:01:11] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:01:52] (03Merged) 10jenkins-bot: wdqs: widen proxy readiness probe timeout/failureThreshold [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345098 (https://phabricator.wikimedia.org/T439248) (owner: 10Gmodena) [09:01:53] (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342963 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [09:02:04] (03PS7) 10JMeybohm: gVisor: use kvm as a platform when hardware virtualization is available [puppet] - 10https://gerrit.wikimedia.org/r/1342962 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [09:03:06] RECOVERY - SSH on netmon2002 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u10 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [09:03:50] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [09:04:50] !log running ncmonitor manually on ncmonitor1001 since last execution failed due to a 504 from MarkMonitor [09:04:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:05:04] (03CR) 10Arnaudb: [C:03+2] deployment_server/k8s: set kubeconfig files for aphlict [puppet] - 10https://gerrit.wikimedia.org/r/1341711 (https://phabricator.wikimedia.org/T436657) (owner: 10Arnaudb) [09:05:14] !log jelto@cumin1004 START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw [09:05:34] (03CR) 10Btullis: [V:03+2 C:03+2] ceph: Move the dummy STS key to the correct hiera file [labs/private] - 10https://gerrit.wikimedia.org/r/1345228 (https://phabricator.wikimedia.org/T435590) (owner: 10Andrea Denisse) [09:09:19] !log jelto@cumin1004 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [09:09:42] RESOLVED: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:09:56] (03PS2) 10Brouberol: site/dse-k8s-worker2006: assign dse_k8s::worker role [puppet] - 10https://gerrit.wikimedia.org/r/1345359 (https://phabricator.wikimedia.org/T439241) [09:09:56] (03PS2) 10Brouberol: site/dse-k8s-worker2007: assign dse_k8s::worker role [puppet] - 10https://gerrit.wikimedia.org/r/1345360 (https://phabricator.wikimedia.org/T439241) [09:09:56] (03PS2) 10Brouberol: site/dse-k8s-worker2008: assign dse_k8s::worker role [puppet] - 10https://gerrit.wikimedia.org/r/1345361 (https://phabricator.wikimedia.org/T439241) [09:09:56] (03PS2) 10Brouberol: site: remove ganeti-jumbo role entries [puppet] - 10https://gerrit.wikimedia.org/r/1345362 (https://phabricator.wikimedia.org/T439241) [09:10:12] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 3.2.23 upgrade (T438828) [09:10:16] T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828 [09:10:24] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 3.2.23 upgrade (T438828) [09:10:28] !log jelto@cumin1004 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [09:10:28] !log jelto@cumin1004 END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw [09:10:33] (03CR) 10JMeybohm: [C:03+1] alertmanager: Repoint serviceops task to the correct project [puppet] - 10https://gerrit.wikimedia.org/r/1344095 (owner: 10RLazarus) [09:10:45] (03PS2) 10Jelto: service::catalog: Set ipip_encapsulation for k8s-ingress-wikikube in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1344197 (https://phabricator.wikimedia.org/T420436) [09:11:21] (03CR) 10Brouberol: [C:03+2] site/dse-k8s-worker2006: assign dse_k8s::worker role [puppet] - 10https://gerrit.wikimedia.org/r/1345359 (https://phabricator.wikimedia.org/T439241) (owner: 10Brouberol) [09:14:01] 06SRE, 10observability, 06Traffic, 13Patch-For-Review: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12368335 (10dkertesz) We might give this another try when we upgrade to 3.4, which marks the SHM version of the stats file as no longer experimental: > MINOR: config:... [09:17:05] FIRING: [4x] PKICertificateExpiry: Intermediate certificate in the trust chain for kafka expires in 29d 3h 49m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [09:19:46] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [09:20:58] !log brouberol@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2006.codfw.wmnet [09:21:18] (03CR) 10Jelto: [C:03+2] service::catalog: Set ipip_encapsulation for k8s-ingress-wikikube in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1344197 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [09:22:26] !log jelto@cumin1004 START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad [09:22:30] (03CR) 10JMeybohm: [C:03+1] "This LGTM. We maybe want this to be added to all mw deployments later." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344653 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake) [09:24:22] 07sre-alert-triage, 06Infrastructure-Foundations: Alert in need of triage: GanetiCACertificateAboutToExpire (instance ganeti6002:9100) - https://phabricator.wikimedia.org/T439379 (10LSobanski) 03NEW [09:24:26] 07sre-alert-triage, 06Infrastructure-Foundations: Alert in need of triage: ExporterUnavailable - https://phabricator.wikimedia.org/T439380 (10LSobanski) 03NEW [09:25:33] (03CR) 10JMeybohm: [C:03+1] dse-k8s: allow unauthenticated reads of the OIDC discovery endpoints [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344034 (https://phabricator.wikimedia.org/T435591) (owner: 10Btullis) [09:26:15] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2006.codfw.wmnet [09:26:21] !log jelto@cumin1004 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [09:27:19] !log brouberol@cumin1004 conftool action : set/weight=10; selector: name=dse-k8s-worker2006.codfw.wmnet [09:27:23] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [09:27:27] !log jelto@cumin1004 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [09:27:27] !log jelto@cumin1004 END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad [09:27:33] !log brouberol@cumin1004 conftool action : set/pooled=yes; selector: name=dse-k8s-worker2006.codfw.wmnet [09:29:49] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [09:31:16] !log brouberol@cumin1004 START - Cookbook sre.hosts.rename from ganeti-jumbo2002 to dse-k8s-worker2007 [09:31:39] !log brouberol@cumin1004 START - Cookbook sre.dns.netbox [09:35:30] !log brouberol@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2002 to dse-k8s-worker2007 - brouberol@cumin1004" [09:35:46] !log brouberol@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2002 to dse-k8s-worker2007 - brouberol@cumin1004" [09:35:46] !log brouberol@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [09:35:46] !log brouberol@cumin1004 START - Cookbook sre.dns.wipe-cache dse-k8s-worker2007 on all recursors [09:35:50] !log brouberol@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2007 on all recursors [09:35:50] !log brouberol@cumin1004 START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2007 [09:36:02] !log brouberol@cumin1004 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2007 [09:36:37] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2002 to dse-k8s-worker2007 [09:39:15] !log brouberol@cumin1004 START - Cookbook sre.hosts.reimage for host dse-k8s-worker2007.codfw.wmnet with OS bookworm [09:40:12] (03PS1) 10Santiago Faci: Deploy GrowthBook 5.1.0 to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345497 (https://phabricator.wikimedia.org/T438954) [09:41:27] (03CR) 10Brouberol: [C:03+1] Deploy GrowthBook 5.1.0 to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345497 (https://phabricator.wikimedia.org/T438954) (owner: 10Santiago Faci) [09:42:08] (03CR) 10Santiago Faci: [C:03+2] Deploy GrowthBook 5.1.0 to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345497 (https://phabricator.wikimedia.org/T438954) (owner: 10Santiago Faci) [09:43:23] (03PS1) 10Jelto: service::catalog: Set ipip_encapsulation for mathoid, mwdebug* in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1345498 (https://phabricator.wikimedia.org/T420436) [09:43:25] (03PS1) 10Jelto: service::catalog: Set ipip_encapsulation for mathoid, mwdebug* in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1345499 (https://phabricator.wikimedia.org/T420436) [09:43:27] jouncebot: nowandnext [09:43:27] No deployments scheduled for the next 0 hour(s) and 16 minute(s) [09:43:27] In 0 hour(s) and 16 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260928T1000) [09:44:54] (03CR) 10Jelto: [C:03+1] "lgtm now" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341335 (https://phabricator.wikimedia.org/T437635) (owner: 10Dzahn) [09:45:00] (03Merged) 10jenkins-bot: Deploy GrowthBook 5.1.0 to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345497 (https://phabricator.wikimedia.org/T438954) (owner: 10Santiago Faci) [09:45:52] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply [09:46:19] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 3.2.23 upgrade (T438828) [09:46:22] T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828 [09:46:30] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply [09:46:46] PROBLEM - Confd vcl based reload on cp6015 is CRITICAL: reload-vcl failed to run since 0h, 2 minutes. https://wikitech.wikimedia.org/wiki/Varnish [09:50:13] !log brouberol@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2007.codfw.wmnet with reason: host reimage [09:52:00] FIRING: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker2007 has a BGP session which is not in the 'established' state. [09:53:06] PROBLEM - Confd vcl based reload on cp6001 is CRITICAL: reload-vcl failed to run since 0h, 2 minutes. https://wikitech.wikimedia.org/wiki/Varnish [09:53:24] (03PS7) 10Clément Goubert: mediawiki: Redirect /api/ to /w/rest.php [deployment-charts] - 10https://gerrit.wikimedia.org/r/1330505 (https://phabricator.wikimedia.org/T433547) [09:53:39] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2007.codfw.wmnet with reason: host reimage [09:54:53] !log jayme@cumin1004 START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker2041.codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [09:54:57] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet [09:55:39] 06SRE, 06Infrastructure-Foundations: Create bookworm-based build host - https://phabricator.wikimedia.org/T379343#12368549 (10MoritzMuehlenhoff) 05Open→03Resolved This is long resolved. In fact by now all central container functions are moved to build2004/trixie (https://phabricator.wikimedia.org/T417389) [09:57:08] RECOVERY - Confd vcl based reload on cp6001 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [09:57:46] RECOVERY - Confd vcl based reload on cp6015 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [09:58:37] (03PS8) 10Clément Goubert: mediawiki: Redirect /api/ to /w/rest.php [deployment-charts] - 10https://gerrit.wikimedia.org/r/1330505 (https://phabricator.wikimedia.org/T433547) [10:00:03] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260928T1000) [10:00:26] !log jayme@cumin1004 START - Cookbook sre.hosts.reimage for host wikikube-worker2041.codfw.wmnet with OS trixie [10:02:24] (03CR) 10Hnowlan: [C:03+2] services_proxy: drop eventgate timeout to 11s [puppet] - 10https://gerrit.wikimedia.org/r/1344739 (https://phabricator.wikimedia.org/T438896) (owner: 10Hnowlan) [10:12:12] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2007.codfw.wmnet with OS bookworm [10:13:28] FIRING: [2x] ProbeDown: Service kafka-logging1003:9093 has failed probes (tcp_kafka_broker_tls_kafka_logging1003_ip4) - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:14:37] (03CR) 10JMeybohm: [C:03+1] mediawiki: Redirect /api/ to /w/rest.php [deployment-charts] - 10https://gerrit.wikimedia.org/r/1330505 (https://phabricator.wikimedia.org/T433547) (owner: 10Clément Goubert) [10:17:52] !log jayme@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage [10:17:58] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 3.2.23 upgrade (T438828) [10:18:01] T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828 [10:19:01] 06SRE, 10observability, 06Traffic, 13Patch-For-Review: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12368616 (10Vgutierrez) 05In progress→03Stalled [10:19:44] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - 3.2.23 upgrade (T438828) [10:19:46] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - 3.2.23 upgrade (T438828) [10:21:45] (03CR) 10Blake: [C:03+2] mw-web: set an anti-affinity preference for thumbor nodes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344653 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake) [10:22:06] (03CR) 10Lucas Werkmeister (WMDE): [C:03+1] "Confirmed unused in wmf.21:" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345130 (https://phabricator.wikimedia.org/T437643) (owner: 10Sbisson) [10:23:54] !log jayme@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage [10:24:34] (03CR) 10Muehlenhoff: [C:03+2] proton: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345488 (owner: 10Muehlenhoff) [10:24:38] (03Merged) 10jenkins-bot: mw-web: set an anti-affinity preference for thumbor nodes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344653 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake) [10:25:10] !log installing chromium security updates [10:25:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:25:53] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-web: apply [10:26:11] !log jmm@deploy1003 helmfile [staging] START helmfile.d/services/proton: apply [10:27:16] !log jmm@deploy1003 helmfile [staging] DONE helmfile.d/services/proton: apply [10:28:20] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-web: apply [10:28:21] !log blake@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-web: apply [10:30:02] !log jmm@deploy1003 helmfile [codfw] START helmfile.d/services/proton: apply [10:30:32] !log blake@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-web: apply [10:30:47] 06SRE, 06DBA, 10Observability-Alerting: single DB server replag / downtime should not page us anymore - https://phabricator.wikimedia.org/T396816#12368654 (10Marostegui) a:05FCeratto-WMF→03None [10:30:54] 14SRE-Sprint-Week-Sustainability-March2023, 06Data-Persistence-Automations, 06DBA, 13Patch-For-Review, 07Sustainability (Incident Followup): Implement (or refactor) a script to move slaves when the master is not available - https://phabricator.wikimedia.org/T196366#12368655 (10Marostegui) a:05FCeratto-W... [10:31:44] (03PS2) 10Filippo Giunchedi: installserver: use cephosd-efi recipe for cloudcephosd105[3-6] [puppet] - 10https://gerrit.wikimedia.org/r/1338879 (https://phabricator.wikimedia.org/T419892) [10:32:49] (03PS3) 10Hnowlan: zuul: migrate icinga checks to prometheus [puppet] - 10https://gerrit.wikimedia.org/r/1343955 (https://phabricator.wikimedia.org/T384939) [10:35:19] (03PS1) 10Marostegui: db1269: Remove note [puppet] - 10https://gerrit.wikimedia.org/r/1345504 [10:35:33] (03CR) 10Marostegui: "This is a noop" [puppet] - 10https://gerrit.wikimedia.org/r/1345504 (owner: 10Marostegui) [10:36:23] (03CR) 10Marostegui: [C:03+2] db1269: Remove note [puppet] - 10https://gerrit.wikimedia.org/r/1345504 (owner: 10Marostegui) [10:38:28] PROBLEM - Host wikikube-worker1260 is DOWN: PING CRITICAL - Packet loss = 80%, RTA = 9304.24 ms [10:39:00] RECOVERY - Host wikikube-worker1260 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [10:41:55] !log jmm@deploy1003 helmfile [codfw] DONE helmfile.d/services/proton: apply [10:42:10] !log jmm@deploy1003 helmfile [eqiad] START helmfile.d/services/proton: apply [10:42:58] !log jayme@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2041.codfw.wmnet with OS trixie [10:43:03] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet [10:43:05] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet [10:43:06] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker2041.codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [10:43:23] !log jmm@deploy1003 helmfile [eqiad] DONE helmfile.d/services/proton: apply [10:49:50] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - 3.2.23 upgrade (T438828) [10:49:54] T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828 [10:51:24] PROBLEM - Host titan1002 is DOWN: PING CRITICAL - Packet loss = 0%, RTA = 2330.95 ms [10:52:14] RECOVERY - Host titan1002 is UP: PING WARNING - Packet loss = 0%, RTA = 1968.49 ms [10:53:28] FIRING: [4x] ProbeDown: Service kafka-logging1003:9093 has failed probes (tcp_kafka_broker_tls_kafka_logging1003_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:54:15] (03CR) 10Muehlenhoff: [C:03+2] Blocklist KCM [puppet] - 10https://gerrit.wikimedia.org/r/1345053 (owner: 10Muehlenhoff) [10:55:41] FIRING: [4x] ProbeDown: Service kafka-logging1003:9093 has failed probes (tcp_kafka_broker_tls_kafka_logging1003_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:55:46] (03PS2) 10Muehlenhoff: Blocklist phonet [puppet] - 10https://gerrit.wikimedia.org/r/1345066 [10:56:19] jouncebot: nowandnext [10:56:20] For the next 0 hour(s) and 3 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260928T1000) [10:56:20] In 2 hour(s) and 3 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260928T1300) [10:56:49] (03CR) 10Hnowlan: "Daniel: I was basing the ipv4 on the fact that zuul::server explicitly uses ipv4 to connect and the behaviour of the old check - happy to " [puppet] - 10https://gerrit.wikimedia.org/r/1343955 (https://phabricator.wikimedia.org/T384939) (owner: 10Hnowlan) [10:57:38] (03CR) 10Hnowlan: "Just to note, `procs_mailman3_web` is not being removed as the current check ensures that a specific number of versions are running rather" [puppet] - 10https://gerrit.wikimedia.org/r/1344226 (https://phabricator.wikimedia.org/T357099) (owner: 10Hnowlan) [10:58:27] (03CR) 10Majavah: [C:04-1] zuul: migrate icinga checks to prometheus (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1343955 (https://phabricator.wikimedia.org/T384939) (owner: 10Hnowlan) [10:58:28] FIRING: [4x] ProbeDown: Service kafka-logging1003:9093 has failed probes (tcp_kafka_broker_tls_kafka_logging1003_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:59:19] (03CR) 10Muehlenhoff: [C:03+2] Blocklist phonet [puppet] - 10https://gerrit.wikimedia.org/r/1345066 (owner: 10Muehlenhoff) [11:01:26] (03PS1) 10Volans: cumin: Add dummy insetup role report config [labs/private] - 10https://gerrit.wikimedia.org/r/1345507 [11:02:21] (03PS1) 10Volans: profile::cumin: Move insetup config to hiera [puppet] - 10https://gerrit.wikimedia.org/r/1345508 [11:04:26] (03CR) 10Brouberol: [C:03+2] site/dse-k8s-worker2007: assign dse_k8s::worker role [puppet] - 10https://gerrit.wikimedia.org/r/1345360 (https://phabricator.wikimedia.org/T439241) (owner: 10Brouberol) [11:06:05] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware), 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12368761 (10fgiunchedi) a:05Jclark-ctr→03fgiunchedi Ok thank you @Jclark-ctr ! I will try poking 1055 and see what happens. In th... [11:07:47] (03CR) 10Zabe: [C:03+2] reconcileTables: Add sleep option [core] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345344 (https://phabricator.wikimedia.org/T438750) (owner: 10Zabe) [11:13:07] (03PS5) 10Btullis: ceph: Enable STS on the codfw rados gateway [puppet] - 10https://gerrit.wikimedia.org/r/1345117 (https://phabricator.wikimedia.org/T435590) [11:14:14] (03CR) 10CI reject: [V:04-1] ceph: Enable STS on the codfw rados gateway [puppet] - 10https://gerrit.wikimedia.org/r/1345117 (https://phabricator.wikimedia.org/T435590) (owner: 10Btullis) [11:14:52] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - 3.2.23 upgrade (T438828) [11:14:55] T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828 [11:15:43] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 3.2.23 upgrade (T438828) [11:15:48] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 3.2.23 upgrade (T438828) [11:16:54] (03Merged) 10jenkins-bot: reconcileTables: Add sleep option [core] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345344 (https://phabricator.wikimedia.org/T438750) (owner: 10Zabe) [11:17:12] !log brouberol@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2007.codfw.wmnet [11:17:16] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1345344|reconcileTables: Add sleep option (T438750)]] [11:17:19] T438750: Clone wbc_entity_usage from local cluster to x1 for all wikidata client wikis - https://phabricator.wikimedia.org/T438750 [11:18:33] (03CR) 10Muehlenhoff: ferm: Absent the NRPE check when migrating from ferm to nftables (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1283620 (owner: 10Muehlenhoff) [11:18:38] (03PS6) 10Muehlenhoff: ferm: Absent the NRPE check when migrating from ferm to nftables [puppet] - 10https://gerrit.wikimedia.org/r/1283620 [11:20:16] !log brouberol@cumin1004 conftool action : set/pooled=yes; selector: name=dse-k8s-worker2007.codfw.wmnet [11:20:23] !log brouberol@cumin1004 conftool action : set/weight=10; selector: name=dse-k8s-worker2007.codfw.wmnet [11:21:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:21:34] !log zabe@deploy1003 zabe: Backport for [[gerrit:1345344|reconcileTables: Add sleep option (T438750)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:22:29] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2007.codfw.wmnet [11:22:46] !log zabe@deploy1003 zabe: Continuing with deployment [11:22:48] !log brouberol@cumin1004 START - Cookbook sre.hosts.rename from ganeti-jumbo2003 to dse-k8s-worker2008 [11:23:11] !log brouberol@cumin1004 START - Cookbook sre.dns.netbox [11:28:09] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345344|reconcileTables: Add sleep option (T438750)]] (duration: 10m 52s) [11:28:12] T438750: Clone wbc_entity_usage from local cluster to x1 for all wikidata client wikis - https://phabricator.wikimedia.org/T438750 [11:28:36] (03CR) 10Majavah: "I am a bit confused why we don't purge `/etc/nagios/nrpe.d` like we do `/usr/local/lib/nagios/plugins`." [puppet] - 10https://gerrit.wikimedia.org/r/1283620 (owner: 10Muehlenhoff) [11:28:42] brouberol@cumin1004 rename (PID 1357772) is awaiting input [11:32:27] (03PS6) 10Btullis: ceph: Enable STS on the codfw rados gateway [puppet] - 10https://gerrit.wikimedia.org/r/1345117 (https://phabricator.wikimedia.org/T435590) [11:33:53] (03PS4) 10Hnowlan: zuul: migrate icinga checks to prometheus [puppet] - 10https://gerrit.wikimedia.org/r/1343955 (https://phabricator.wikimedia.org/T384939) [11:35:00] (03CR) 10Hnowlan: zuul: migrate icinga checks to prometheus (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1343955 (https://phabricator.wikimedia.org/T384939) (owner: 10Hnowlan) [11:35:29] (03CR) 10Volans: "Related to I0c3f97d7f44097f6e503529c8d72472ae1790282" [labs/private] - 10https://gerrit.wikimedia.org/r/1345507 (owner: 10Volans) [11:35:39] (03CR) 10Volans: "Related to I0c3f97d7f44097f6e503529c8d72472ae1790282" [puppet] - 10https://gerrit.wikimedia.org/r/1345508 (owner: 10Volans) [11:42:03] !log brouberol@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2003 to dse-k8s-worker2008 - brouberol@cumin1004" [11:42:16] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345117 (https://phabricator.wikimedia.org/T435590) (owner: 10Btullis) [11:42:44] (03CR) 10Brouberol: [C:03+1] ceph: Enable STS on the codfw rados gateway [puppet] - 10https://gerrit.wikimedia.org/r/1345117 (https://phabricator.wikimedia.org/T435590) (owner: 10Btullis) [11:42:59] !log brouberol@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2003 to dse-k8s-worker2008 - brouberol@cumin1004" [11:42:59] !log brouberol@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [11:42:59] !log brouberol@cumin1004 START - Cookbook sre.dns.wipe-cache dse-k8s-worker2008 on all recursors [11:43:02] !log brouberol@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2008 on all recursors [11:43:03] !log brouberol@cumin1004 START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2008 [11:43:15] !log brouberol@cumin1004 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2008 [11:43:16] (03CR) 10Btullis: [C:03+2] dse-k8s: allow unauthenticated reads of the OIDC discovery endpoints [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344034 (https://phabricator.wikimedia.org/T435591) (owner: 10Btullis) [11:43:51] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2003 to dse-k8s-worker2008 [11:43:51] (03PS1) 10Blake: mw-web-next: increase replica count to 2. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345513 (https://phabricator.wikimedia.org/T439396) [11:46:12] !log brouberol@cumin1004 START - Cookbook sre.hosts.reimage for host dse-k8s-worker2008.codfw.wmnet with OS bookworm [11:47:36] (03CR) 10Muehlenhoff: "> I am a bit confused why we don't purge /etc/nagios/nrpe.d like we do /usr/local/lib/nagios/plugins." [puppet] - 10https://gerrit.wikimedia.org/r/1283620 (owner: 10Muehlenhoff) [11:48:34] (03Merged) 10jenkins-bot: dse-k8s: allow unauthenticated reads of the OIDC discovery endpoints [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344034 (https://phabricator.wikimedia.org/T435591) (owner: 10Btullis) [11:49:10] (03CR) 10Clément Goubert: [C:03+1] mw-web-next: increase replica count to 2. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345513 (https://phabricator.wikimedia.org/T439396) (owner: 10Blake) [11:49:51] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 3.2.23 upgrade (T438828) [11:49:54] T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828 [11:50:11] (03CR) 10Blake: [C:03+2] mw-web-next: increase replica count to 2. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345513 (https://phabricator.wikimedia.org/T439396) (owner: 10Blake) [11:52:43] (03Merged) 10jenkins-bot: mw-web-next: increase replica count to 2. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345513 (https://phabricator.wikimedia.org/T439396) (owner: 10Blake) [11:53:02] (03PS2) 101Veertje: Add daily background rotation with attribution and credit fixes [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1306065 [11:53:06] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-web: apply [11:53:20] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-web: apply [11:53:21] !log blake@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-web: apply [11:53:42] !log blake@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-web: apply [11:53:46] (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342962 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [11:53:50] (03CR) 10Majavah: "Hmm, the ms-backup1003 alert is not an Icinga check. It seems to be a nrpe2nodexp Prometheus check provisioned by the `nrpe::monitor_servi" [puppet] - 10https://gerrit.wikimedia.org/r/1283620 (owner: 10Muehlenhoff) [11:53:57] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [11:54:10] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [11:57:25] !log brouberol@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2008.codfw.wmnet with reason: host reimage [11:57:27] (03CR) 10Muehlenhoff: [C:03+2] Remove cumin1003 from config for private Homer [puppet] - 10https://gerrit.wikimedia.org/r/1345088 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [11:59:00] FIRING: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker2008 has a BGP session which is not in the 'established' state. [12:04:27] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2008.codfw.wmnet with reason: host reimage [12:05:13] (03PS3) 10Filippo Giunchedi: installserver: use cephosd-efi recipe for cloudcephosd105[3-6] [puppet] - 10https://gerrit.wikimedia.org/r/1338879 (https://phabricator.wikimedia.org/T419892) [12:06:42] FIRING: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:08:11] (03CR) 10Filippo Giunchedi: [V:03+2 C:03+2] installserver: use cephosd-efi recipe for cloudcephosd105[3-6] [puppet] - 10https://gerrit.wikimedia.org/r/1338879 (https://phabricator.wikimedia.org/T419892) (owner: 10Filippo Giunchedi) [12:12:13] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 28 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344771 (https://phabricator.wikimedia.org/T436332) (owner: 10VolkerE) [12:12:55] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 28 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344772 (https://phabricator.wikimedia.org/T436332) (owner: 10VolkerE) [12:14:19] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply [12:14:22] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply [12:14:55] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply [12:15:28] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply [12:15:50] (03PS13) 10Arnaudb: aphlict: add chart for the Phabricator notification relay [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341856 (https://phabricator.wikimedia.org/T436657) [12:16:42] RESOLVED: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:17:23] (03PS3) 10Brouberol: site/dse-k8s-worker2008: assign dse_k8s::worker role [puppet] - 10https://gerrit.wikimedia.org/r/1345361 (https://phabricator.wikimedia.org/T439241) [12:17:23] (03PS3) 10Brouberol: site: remove ganeti-jumbo role entries [puppet] - 10https://gerrit.wikimedia.org/r/1345362 (https://phabricator.wikimedia.org/T439241) [12:17:24] (03PS2) 10Muehlenhoff: Remove cumin1003 from the alertmanager and tcpircbot configs [puppet] - 10https://gerrit.wikimedia.org/r/1345087 (https://phabricator.wikimedia.org/T427897) [12:17:31] !log btullis@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [12:17:49] !log btullis@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [12:18:06] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [12:18:26] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [12:19:08] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 3.2.23 upgrade (T438828) [12:19:11] T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828 [12:19:24] (03CR) 10Brouberol: [C:03+2] site/dse-k8s-worker2008: assign dse_k8s::worker role [puppet] - 10https://gerrit.wikimedia.org/r/1345361 (https://phabricator.wikimedia.org/T439241) (owner: 10Brouberol) [12:19:27] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [12:19:58] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [12:21:57] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2008.codfw.wmnet with OS bookworm [12:23:01] (03PS1) 10Ozge: httpbb(liftwing): Update Lift Wing suites and add a runner [puppet] - 10https://gerrit.wikimedia.org/r/1345534 (https://phabricator.wikimedia.org/T439385) [12:23:27] (03PS2) 10Ozge: httpbb(liftwing): Update Lift Wing suites and add a runner [puppet] - 10https://gerrit.wikimedia.org/r/1345534 (https://phabricator.wikimedia.org/T439385) [12:24:58] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [12:26:10] (03CR) 10CI reject: [V:04-1] httpbb(liftwing): Update Lift Wing suites and add a runner [puppet] - 10https://gerrit.wikimedia.org/r/1345534 (https://phabricator.wikimedia.org/T439385) (owner: 10Ozge) [12:27:07] (03CR) 10Btullis: [C:03+2] ceph: Enable STS on the codfw rados gateway [puppet] - 10https://gerrit.wikimedia.org/r/1345117 (https://phabricator.wikimedia.org/T435590) (owner: 10Btullis) [12:27:49] !log brouberol@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2008.codfw.wmnet [12:28:18] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [12:29:10] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [12:29:49] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [12:29:58] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [12:33:05] !log brouberol@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2008.codfw.wmnet [12:33:12] !log brouberol@cumin1004 conftool action : set/weight=10; selector: name=dse-k8s-worker2008.codfw.wmnet [12:33:21] !log brouberol@cumin1004 conftool action : set/pooled=yes; selector: name=dse-k8s-worker2008.codfw.wmnet [12:34:17] (03PS1) 10Giuseppe Lavagetto: shellbox: allow inotify probes to be set for tls certs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345535 (https://phabricator.wikimedia.org/T439409) [12:34:58] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [12:36:42] (03CR) 10CI reject: [V:04-1] shellbox: allow inotify probes to be set for tls certs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345535 (https://phabricator.wikimedia.org/T439409) (owner: 10Giuseppe Lavagetto) [12:37:00] (03CR) 10Brouberol: [C:03+2] site: remove ganeti-jumbo role entries [puppet] - 10https://gerrit.wikimedia.org/r/1345362 (https://phabricator.wikimedia.org/T439241) (owner: 10Brouberol) [12:43:00] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review: Adapt profile::nginx to new packaging scheme introduced in Bookworm - https://phabricator.wikimedia.org/T329529#12369253 (10MoritzMuehlenhoff) [12:43:04] (03PS2) 10Giuseppe Lavagetto: shellbox: allow inotify probes to be set for tls certs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345535 (https://phabricator.wikimedia.org/T439409) [12:43:12] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review: Adapt profile::nginx to new packaging scheme introduced in Bookworm - https://phabricator.wikimedia.org/T329529#12369254 (10MoritzMuehlenhoff) [12:44:58] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [12:45:16] (03CR) 10CI reject: [V:04-1] shellbox: allow inotify probes to be set for tls certs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345535 (https://phabricator.wikimedia.org/T439409) (owner: 10Giuseppe Lavagetto) [12:49:58] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [12:50:13] (03PS1) 10Brouberol: dse-k8s-eqiad: define the postgresql-growthbook-fundraising ns [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345537 (https://phabricator.wikimedia.org/T438350) [12:50:15] (03PS1) 10Brouberol: dse-k8s-eqiad: define the postgresql-growthbook-fundraising release [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345538 (https://phabricator.wikimedia.org/T438350) [12:50:56] (03PS2) 10Brouberol: dse-k8s-eqiad: define the postgresql-growthbook-fundraising release [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345538 (https://phabricator.wikimedia.org/T438350) [12:52:56] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T439299#12369286 (10Jclark-ctr) a:03Jclark-ctr [12:52:57] (03PS1) 10Brouberol: deployment_server: define the postgresql-growthbook-fundraising kubeconfigs [puppet] - 10https://gerrit.wikimedia.org/r/1345540 (https://phabricator.wikimedia.org/T438350) [12:53:51] (03CR) 10Elukey: [C:03+1] profile::cumin: Move insetup config to hiera [puppet] - 10https://gerrit.wikimedia.org/r/1345508 (owner: 10Volans) [12:54:08] (03CR) 10Elukey: [C:03+1] cumin: Add dummy insetup role report config [labs/private] - 10https://gerrit.wikimedia.org/r/1345507 (owner: 10Volans) [12:54:58] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [12:56:47] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-e8-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439340#12369301 (10Jclark-ctr) ps1-e8-eqiad.mgmt.eqiad.wmnet #1: Sensor: Phase, AA:L1-L2, Active Power Value: 1.691 kW (power) Thresholds: High: 1650 #2: Sensor: Phase,... [12:57:17] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review: Adapt profile::nginx to new packaging scheme introduced in Bookworm - https://phabricator.wikimedia.org/T329529#12369302 (10MoritzMuehlenhoff) [12:59:10] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-e8-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439340#12369319 (10Jclark-ctr) 05Open→03Resolved a:03Jclark-ctr [12:59:58] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: gettimeofday() says it's time for UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260928T1300) [13:00:05] stephanebisson, Ameisenigel, and Volker_E: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:23] o/ [13:00:43] o/ [13:00:57] stephanebisson: want to start with your change? [13:01:13] Lucas_WMDE yes [13:01:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:01:27] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:02:10] (03PS1) 10Brouberol: Define the postgresql-growthbook-fundraising discovery record [dns] - 10https://gerrit.wikimedia.org/r/1345543 (https://phabricator.wikimedia.org/T438350) [13:02:27] Ready for backport [13:02:33] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbisson@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345130 (https://phabricator.wikimedia.org/T437643) (owner: 10Sbisson) [13:02:55] Ameisenigel: hi! stephanebisson is deploying first, then we can do yours [13:03:33] (03Merged) 10jenkins-bot: Remove unused config: ArticleGuidanceRedirectEntryPointTitles [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345130 (https://phabricator.wikimedia.org/T437643) (owner: 10Sbisson) [13:03:46] !log sbisson@deploy1003 Started scap sync-world: Backport for [[gerrit:1345130|Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] [13:03:49] T437643: Remove ArticleGuidanceRedirectEntryPointTitles - https://phabricator.wikimedia.org/T437643 [13:04:58] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:06:11] (03PS3) 10Brouberol: dse-k8s-eqiad: define the postgresql-growthbook-fundraising release [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345538 (https://phabricator.wikimedia.org/T438350) [13:06:26] Lucas_WMDE Thanks, no hurry [13:07:45] !log sbisson@deploy1003 sbisson: Backport for [[gerrit:1345130|Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:08:01] (03PS4) 10Brouberol: dse-k8s-eqiad: define the postgresql-growthbook-fundraising release [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345538 (https://phabricator.wikimedia.org/T438350) [13:08:26] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, October 05 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343917 (https://phabricator.wikimedia.org/T437577) (owner: 10Seanleong-wmde) [13:09:34] here as Volker_E38 today, ready for backport [13:10:04] Lucas_WMDE have you seen this error before? https://spiderpig.wikimedia.org/jobs/2873#log [13:10:37] (03PS3) 10Ozge: httpbb(liftwing): Update Lift Wing suites and add a runner [puppet] - 10https://gerrit.wikimedia.org/r/1345534 (https://phabricator.wikimedia.org/T439385) [13:11:14] Well, it worked after a retry. [13:11:23] !log sbisson@deploy1003 sbisson: Continuing with deployment [13:12:00] * Lucas_WMDE looks [13:12:18] yeah I think that’s okay to retry [13:14:58] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:15:44] (03PS1) 10Brouberol: dse-k8s-codfw: dse-k8s-worker200[6-8] to the ingress svc and cluster nodes [puppet] - 10https://gerrit.wikimedia.org/r/1345548 (https://phabricator.wikimedia.org/T439220) [13:16:41] (03CR) 10FNegri: [C:03+1] "Five backups sounds fine to me." [puppet] - 10https://gerrit.wikimedia.org/r/1344255 (https://phabricator.wikimedia.org/T438731) (owner: 10Majavah) [13:16:44] (03PS2) 10Atsuko: airflow-test-k8s: upgrade to airflow 3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345077 (https://phabricator.wikimedia.org/T433383) [13:16:44] !log sbisson@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345130|Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)]] (duration: 12m 58s) [13:16:47] T437643: Remove ArticleGuidanceRedirectEntryPointTitles - https://phabricator.wikimedia.org/T437643 [13:16:55] I'm done [13:17:22] Over to you Lucas_WMDE [13:17:45] alright, thanks [13:18:05] probably safe enough to deploy those two config changes by Ameisenigel together [13:18:25] !log jclark@cumin1004 START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:18:36] (03PS1) 10Bking: Revert "Cirrussearch: Enable performance governor on Row D hosts" [puppet] - 10https://gerrit.wikimedia.org/r/1345549 [13:18:59] Yeah, I think so [13:19:14] (03PS2) 10Bking: Revert "Cirrussearch: Enable performance governor on Row D hosts" [puppet] - 10https://gerrit.wikimedia.org/r/1345549 [13:19:18] (03CR) 10Bking: [V:03+2 C:03+2] Revert "Cirrussearch: Enable performance governor on Row D hosts" [puppet] - 10https://gerrit.wikimedia.org/r/1345549 (owner: 10Bking) [13:19:24] * Lucas_WMDE looks more closely at https://phabricator.wikimedia.org/T386776 [13:19:26] !log jclark@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:19:29] what on earth happened there [13:19:58] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:20:06] The mess in the task or the reverted patch? [13:20:12] (03PS4) 10Ozge: httpbb(liftwing): Update Lift Wing suites and add a runner [puppet] - 10https://gerrit.wikimedia.org/r/1345534 (https://phabricator.wikimedia.org/T439385) [13:20:22] the task [13:20:32] though I’m now trying to figure out how the current patch is different from the reverted one [13:20:47] Last time database tables have not been created in advance, this was now done Zabe [13:21:03] aha, okay [13:21:37] and then it took over a year for someone to try again? [13:21:45] !log jclark@cumin1004 START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:21:51] I guess that’s enough punishment for using LLMs to write task descriptions, and I can stomach the thought of deploying the new change /hj [13:21:57] anyway thanks for picking that up [13:22:07] but also in light of this it seems safer to deploy the changes separately after all ^^ [13:22:18] !log jclark@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:22:21] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345052 (https://phabricator.wikimedia.org/T439159) (owner: 10Ameisenigel) [13:22:23] Sure [13:22:26] so let’s start with just FlaggedRevs [13:22:51] (03CR) 10CI reject: [V:04-1] httpbb(liftwing): Update Lift Wing suites and add a runner [puppet] - 10https://gerrit.wikimedia.org/r/1345534 (https://phabricator.wikimedia.org/T439385) (owner: 10Ozge) [13:22:59] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review, 07Upstream: Ignore Valid-Until for WMF container images - https://phabricator.wikimedia.org/T438866#12369414 (10MoritzMuehlenhoff) 05Open→03Resolved Valid-Until is now disabled for the base images (and production images built on top). For the... [13:23:03] (03CR) 10Brouberol: [C:03+1] airflow-test-k8s: upgrade to airflow 3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345077 (https://phabricator.wikimedia.org/T433383) (owner: 10Atsuko) [13:23:04] * Lucas_WMDE looks if FR has tables [13:23:16] oh wait, this was only one more namespace right [13:23:17] (03Merged) 10jenkins-bot: Enable FlaggedRevs for the Author NS on ruwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345052 (https://phabricator.wikimedia.org/T439159) (owner: 10Ameisenigel) [13:23:27] !log lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1345052|Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] [13:23:31] T439159: Enable FlaggedRevs for the Author NS on ruwikisource - https://phabricator.wikimedia.org/T439159 [13:23:37] Yes, FR is already enable [13:23:46] (03PS4) 101Veertje: Fix login form: responsive labels and mobile-friendly layout [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 [13:24:03] yeah ok [13:24:10] (03CR) 101Veertje: "Done" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 (owner: 101Veertje) [13:27:10] (03PS5) 10Ozge: httpbb(liftwing): Update Lift Wing suites and add a runner [puppet] - 10https://gerrit.wikimedia.org/r/1345534 (https://phabricator.wikimedia.org/T439385) [13:27:15] hm, k8s deployment progress (sync-testservers-k8s) seems to be standing still for a bit… [13:27:23] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:27:28] !log lucaswerkmeister-wmde@deploy1003 ameisenigel, lucaswerkmeister-wmde: Backport for [[gerrit:1345052|Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:27:33] Ameisenigel: please test :) [13:29:12] (03CR) 10Andrew Bogott: [C:03+2] cloudvps-project-usage.py: Add a few quota metrics [puppet] - 10https://gerrit.wikimedia.org/r/1343670 (https://phabricator.wikimedia.org/T436275) (owner: 10Andrew Bogott) [13:29:43] Looks good, we can go on [13:29:53] (03CR) 10Ssingh: "Let's rebase this and get this merged?" [alerts] - 10https://gerrit.wikimedia.org/r/1217262 (owner: 10CDobbins) [13:29:58] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:30:16] !log lucaswerkmeister-wmde@deploy1003 ameisenigel, lucaswerkmeister-wmde: Continuing with deployment [13:30:17] ok, thanks! [13:30:40] !log dpogorzelski@cumin1004 conftool action : set/pooled=true; selector: dnsdisc=inference,name=codfw [13:30:53] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345551 [13:32:06] !log dpogorzelski@cumin1004 conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-serve,name=codfw [13:33:26] !log depool cp7001 for some purged tests - T439361 [13:33:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:33:29] T439361: Reduce goroutine churn in purged - https://phabricator.wikimedia.org/T439361 [13:33:30] !log filippo@cumin1004 START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm [13:33:40] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12369444 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1004 for host cloudcephosd1055.eqiad.wmnet with OS bookworm [13:35:30] !log lucaswerkmeister-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345052|Enable FlaggedRevs for the Author NS on ruwikisource (T439159)]] (duration: 12m 02s) [13:35:33] T439159: Enable FlaggedRevs for the Author NS on ruwikisource - https://phabricator.wikimedia.org/T439159 [13:35:51] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344216 (https://phabricator.wikimedia.org/T386776) (owner: 10Ameisenigel) [13:36:29] (03CR) 10Bking: [C:03+1] dse-k8s-codfw: dse-k8s-worker200[6-8] to the ingress svc and cluster nodes [puppet] - 10https://gerrit.wikimedia.org/r/1345548 (https://phabricator.wikimedia.org/T439220) (owner: 10Brouberol) [13:36:42] !log dpogorzelski@cumin1004 conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-staging,name=codfw [13:36:50] (03CR) 10Brouberol: [C:03+2] dse-k8s-codfw: dse-k8s-worker200[6-8] to the ingress svc and cluster nodes [puppet] - 10https://gerrit.wikimedia.org/r/1345548 (https://phabricator.wikimedia.org/T439220) (owner: 10Brouberol) [13:37:19] (03Merged) 10jenkins-bot: Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344216 (https://phabricator.wikimedia.org/T386776) (owner: 10Ameisenigel) [13:37:32] !log lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1344216|Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] [13:37:35] T386776: Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org - https://phabricator.wikimedia.org/T386776 [13:39:10] (03CR) 10Ozge: "- I've created a run_all.sh." [puppet] - 10https://gerrit.wikimedia.org/r/1345534 (https://phabricator.wikimedia.org/T439385) (owner: 10Ozge) [13:39:24] (03CR) 10Majavah: [C:03+2] P:wmcs::etcd::backup: Cleanup older backups [puppet] - 10https://gerrit.wikimedia.org/r/1344255 (https://phabricator.wikimedia.org/T438731) (owner: 10Majavah) [13:39:29] 10ops-codfw, 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Enable Performance Governor on Cirrussearch hosts - https://phabricator.wikimedia.org/T438877#12369472 (10bking) 05In progress→03Declined Apologies for the delay. I've reverted the performance governor chang... [13:39:58] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:41:23] (03PS3) 10Ssingh: services: make urldownloader probe paging [puppet] - 10https://gerrit.wikimedia.org/r/1328218 (https://phabricator.wikimedia.org/T435648) [13:41:36] !log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, ameisenigel: Backport for [[gerrit:1344216|Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:41:38] !log jclark@cumin1004 START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:41:44] Ameisenigel: please test! [13:42:09] ruh roh, exception [13:42:15] !log jclark@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:42:20] to the logstashmobile! [13:42:56] Missing field: page_lang [13:42:58] from LinkCache [13:43:33] I wonder if some cache needs to be purged? [13:44:03] (but AFAIK LinkCache is usually in-process) [13:44:24] (03CR) 10Bking: [C:03+2] dse-k8s: add opensearch-semantic-search-ssd records [dns] - 10https://gerrit.wikimedia.org/r/1345160 (https://phabricator.wikimedia.org/T438058) (owner: 10Ebernhardson) [13:44:37] I take a look [13:44:40] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for krb1002.mgmt:22 - https://phabricator.wikimedia.org/T439280#12369499 (10Jclark-ctr) Device was sitting at SystemRescue. It looks like the BIOS settings had been changed to UEFI, so I reverted them back to Legacy. It is now at the GRUB prompt. [13:44:51] !log bking@dns2004 START - running authdns-update [13:44:58] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:46:27] apparently the last time a wiki gained Translate powers was T420656, and that had some other issues, but none with this message AFAICT [13:46:28] T420656: Enable Translate extension for Abstract Wikipedia - https://phabricator.wikimedia.org/T420656 [13:47:04] pinging James_F anyway in case he’s around and remembers – did you encounter “InvalidArgumentException: Missing field: page_lang” when enabling Translate on abstractwiki? [13:47:19] !log bking@dns2004 END - running authdns-update [13:48:30] (03CR) 10Muehlenhoff: "Sounds good to me, with all the revised changes on the wikitech page this seems ready to be enabled" [puppet] - 10https://gerrit.wikimedia.org/r/1328218 (https://phabricator.wikimedia.org/T435648) (owner: 10Ssingh) [13:48:45] It seems like it is not only needed to create the database tables, but we also need to update the existing tables [13:48:46] aha, https://phabricator.wikimedia.org/T205349#7466017 has it (from 2021!) [13:49:41] (03PS1) 10Giuseppe Lavagetto: shellbox: enable polling under gVisor for TLS certificate changes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345556 (https://phabricator.wikimedia.org/T439409) [13:50:37] but that task sounds like it should have been fixed, there’s an attached Gerrit change… [13:50:43] Ameisenigel: any idea how to do that? (also, source?) [13:51:11] [username changed once more due to hopping mobile networks] [13:51:51] !log Rebalancing is in progress on kafka-logging in eqiad to evacuate kafka-logging1001 and kafka-logging1002 - T432444 [13:51:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:51:55] T432444: Provision kafka-logging100[6-8] and kafka-logging200[6-8] - https://phabricator.wikimedia.org/T432444 [13:51:56] jouncebot: next [13:51:56] In 0 hour(s) and 38 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260928T1430) [13:52:59] (03PS1) 10Dpogorzelski: liftwing-studio: update the docs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345558 (https://phabricator.wikimedia.org/T438347) [13:53:14] (03CR) 10Dpogorzelski: [C:03+2] liftwing-studio: update the docs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345558 (https://phabricator.wikimedia.org/T438347) (owner: 10Dpogorzelski) [13:53:16] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] liftwing-studio: update the docs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345558 (https://phabricator.wikimedia.org/T438347) (owner: 10Dpogorzelski) [13:53:36] I think we have to abort this deploy and revert the config change [13:53:41] and then hopefully someone can figure out what’s wrong [13:53:44] !log jclark@cumin1004 START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:53:52] but there’s not enough time left in the window to try to fix this IMHO [13:54:14] !log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, ameisenigel: Rolling back deployment [13:54:31] !log jclark@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:56:26] (03CR) 10Ssingh: [C:03+2] services: make urldownloader probe paging [puppet] - 10https://gerrit.wikimedia.org/r/1328218 (https://phabricator.wikimedia.org/T435648) (owner: 10Ssingh) [13:57:03] (03PS1) 10Lucas Werkmeister (WMDE): Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345559 (https://phabricator.wikimedia.org/T386776) [13:57:11] !log lucaswerkmeister-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344216|Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)]] (duration: 19m 38s) [13:57:14] T386776: Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org - https://phabricator.wikimedia.org/T386776 [13:57:19] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345559 (https://phabricator.wikimedia.org/T386776) (owner: 10Lucas Werkmeister (WMDE)) [13:57:21] !log jclark@cumin1004 START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [13:57:46] Lucas_WMDE: Sorry, am on a flight. It sounds vaguely familiar, but for wikifunctions.org a few years ago not abstract. [13:57:59] !log repool cp7001 - T439361 [13:58:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:58:02] T439361: Reduce goroutine churn in purged - https://phabricator.wikimedia.org/T439361 [13:58:16] (03Merged) 10jenkins-bot: Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345559 (https://phabricator.wikimedia.org/T386776) (owner: 10Lucas Werkmeister (WMDE)) [13:58:29] !log lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1345559|Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] [13:58:33] The problem seems to be https://www.mediawiki.org/wiki/Manual:Page_table#page_lang and I think this is not touched by createExtensionTables.php but I will need more time to look into this more detailed [13:58:56] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for krb1002.mgmt:22 - https://phabricator.wikimedia.org/T439280#12369585 (10Jclark-ctr) Found at some point was removed from rack. wrong network cable was plugged back in. Reset bmc and provisioned again [14:00:36] James_F: alright, thanks – have a good flight! if you remember any more, it might be helpful at https://phabricator.wikimedia.org/T386776, but if not it’s also okay of course [14:02:06] (03PS1) 10Ilias Sarantopoulos: ml-services: expose gpt-oss-safeguard-20b publicly [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345562 (https://phabricator.wikimedia.org/T439395) [14:02:41] !log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde: Backport for [[gerrit:1345559|Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:02:45] T386776: Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org - https://phabricator.wikimedia.org/T386776 [14:02:56] !log jclark@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [14:03:51] seems to be healthy again, continuing [14:03:53] !log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde: Continuing with deployment [14:05:27] !log jclark@cumin1004 START - Cookbook sre.dns.netbox [14:09:10] !log lucaswerkmeister-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345559|Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)]] (duration: 10m 40s) [14:09:13] T386776: Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org - https://phabricator.wikimedia.org/T386776 [14:09:18] !log jclark@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding krb1002 to eqiad - jclark@cumin1004" [14:09:41] !log jclark@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding krb1002 to eqiad - jclark@cumin1004" [14:09:41] !log jclark@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:10:06] !log jclark@cumin1004 START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [14:10:11] ok, that’s done… [14:10:28] Volker_E2: how urgent are those two backports? the window is over but there’s a 20-minute break until the next one [14:10:38] !log jclark@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [14:10:44] (03CR) 10Bking: [C:03+2] dse-k8s: Add LVS/services proxy config for new service [puppet] - 10https://gerrit.wikimedia.org/r/1345171 (https://phabricator.wikimedia.org/T438058) (owner: 10Bking) [14:12:11] Lucas_WMDE Our experiment is starting tomorrow, and we will not only need to fit in these two, but two more later or tomorrow. The earlier the better :) [14:12:17] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for krb1002.mgmt:22 - https://phabricator.wikimedia.org/T439280#12369658 (10Jclark-ctr) Leaving ticket open tilll T435354 is resolved [14:12:19] ok [14:12:23] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for krb1002.mgmt:22 - https://phabricator.wikimedia.org/T439280#12369660 (10Jclark-ctr) a:03Jclark-ctr [14:12:24] do you need a deployer or can you deploy yourself? [14:12:54] I'd need a deployer, as I debug SpiderPig access currently (something changed since Feb :} ) [14:12:59] ok [14:13:14] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344771 (https://phabricator.wikimedia.org/T436332) (owner: 10VolkerE) [14:13:14] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344772 (https://phabricator.wikimedia.org/T436332) (owner: 10VolkerE) [14:14:41] (03Abandoned) 10Urbanecm: ReassignMentees: Actually deduplicate reassignMenteesJob [extensions/GrowthExperiments] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344713 (https://phabricator.wikimedia.org/T418194) (owner: 10Urbanecm) [14:14:43] Lucas_WMDE Appreciated! [14:14:58] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:15:45] Lucas_WMDE #til that one could even use the deployment window break until the next one. [14:15:57] I mean, not officially AFAIK [14:16:03] but I don’t think I’ve been yelled at for it yet ;) [14:17:13] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [14:17:58] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [14:19:36] (03Merged) 10jenkins-bot: feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344771 (https://phabricator.wikimedia.org/T436332) (owner: 10VolkerE) [14:19:58] FIRING: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:20:30] (03Merged) 10jenkins-bot: feat(AccountSetup), styles: Apply correct background and button styles [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344772 (https://phabricator.wikimedia.org/T436332) (owner: 10VolkerE) [14:20:47] !log lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1344771|feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772|feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] [14:20:51] T436332: Recommended reading exp: Final design review and adjustments for recommended articles - https://phabricator.wikimedia.org/T436332 [14:21:46] filippo@cumin1004 reimage (PID 1377705) is awaiting input [14:24:53] !log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, volker-e: Backport for [[gerrit:1344771|feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772|feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:25:03] Volker_E2: please test ^^ [14:27:11] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [14:29:58] RESOLVED: [2x] CertAlmostExpired: Certificate for service shellbox-timeline:4012 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#shellbox-timeline:4012 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:30:06] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260928T1430) [14:30:39] Volker_E2: can you test the change on WikimediaDebug? (or is it untestable?) [14:36:46] 06SRE, 06Infrastructure-Foundations: Disable TUN/TAP outside of virtualisation roles - https://phabricator.wikimedia.org/T438459#12369731 (10MoritzMuehlenhoff) p:05Triage→03Medium a:03MoritzMuehlenhoff [14:37:37] (03CR) 10Ilias Sarantopoulos: ml-services: expose gpt-oss-safeguard-20b publicly (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345562 (https://phabricator.wikimedia.org/T439395) (owner: 10Ilias Sarantopoulos) [14:37:51] 06SRE, 06Infrastructure-Foundations: Upgrade rpki hosts to Trixie - https://phabricator.wikimedia.org/T438122#12369734 (10MoritzMuehlenhoff) p:05Triage→03Medium [14:40:12] Volker_E2: please test the backports on WikimediaDebug :) [14:40:20] PROBLEM - Swift https backend on ms-fe1016 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Swift [14:41:10] RECOVERY - Swift https backend on ms-fe1016 is OK: HTTP OK: HTTP/1.1 200 OK - 567 bytes in 0.061 second response time https://wikitech.wikimedia.org/wiki/Swift [14:42:53] (03CR) 10JMeybohm: "Quick question: Does this really require it's own chart or could it use something generic like python-webapp?" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341856 (https://phabricator.wikimedia.org/T436657) (owner: 10Arnaudb) [14:43:49] Lucas_WMDE is it next or experimental? [14:44:04] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439320#12369768 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [14:44:08] k8s-mwdebug [14:44:13] ok [14:45:56] 10ops-eqiad, 06SRE, 06DC-Ops, 06ServiceOps: Q#:rack/setup/install 26 wikikube hosts - https://phabricator.wikimedia.org/T439427 (10RobH) 03NEW [14:46:38] Lucas_WMDE as expected! [14:46:59] 10ops-eqiad, 06SRE, 06DC-Ops, 06ServiceOps: Q#:rack/setup/install 26 wikikube hosts - https://phabricator.wikimedia.org/T439427#12369797 (10RobH) a:03Clement_Goubert @Clement_Goubert, Please review and update this rack task. This task requires update for all hostnames, racking location, and checklist e... [14:47:07] 10ops-eqiad, 06SRE, 06DC-Ops, 06ServiceOps: Q#:rack/setup/install 26 wikikube hosts - https://phabricator.wikimedia.org/T439427#12369805 (10RobH) [14:47:42] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip_encapsulation for mathoid, mwdebug* in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1345498 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [14:47:54] 07sre-alert-triage, 06Infrastructure-Foundations: Alert in need of triage: ExporterUnavailable - https://phabricator.wikimedia.org/T439380#12369809 (10MoritzMuehlenhoff) p:05Triage→03Medium [14:48:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:48:13] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip_encapsulation for mathoid, mwdebug* in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1345499 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [14:48:15] 07sre-alert-triage, 06Infrastructure-Foundations: Alert in need of triage: GanetiCACertificateAboutToExpire (instance ganeti6002:9100) - https://phabricator.wikimedia.org/T439379#12369811 (10MoritzMuehlenhoff) p:05Triage→03High a:03MoritzMuehlenhoff [14:49:07] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps: Q3:rack/setup/install 8 new wikikube hosts - https://phabricator.wikimedia.org/T439428 (10RobH) 03NEW [14:49:07] Volker_E2: okay, thanks! [14:49:10] sorry, was distracted for a moment [14:49:11] !log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, volker-e: Continuing with deployment [14:49:41] (03PS1) 10Tiziano Fogli: aptrepo/updates: remove upper bound on Grafana version [puppet] - 10https://gerrit.wikimedia.org/r/1345569 (https://phabricator.wikimedia.org/T438454) [14:49:50] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps: Q3:rack/setup/install 8 new wikikube hosts - https://phabricator.wikimedia.org/T439428#12369832 (10RobH) a:03Clement_Goubert @Clement_Goubert, Please review and update this rack task. This task requires update for all hostnames, racking location, and checklis... [14:50:06] 10ops-eqiad, 06SRE, 06DC-Ops, 06ServiceOps: Q3:rack/setup/install 26 wikikube hosts - https://phabricator.wikimedia.org/T439427#12369835 (10RobH) [14:50:52] 06SRE, 06ServiceOps: 26 wikikube in eqiad imlementation tracking - https://phabricator.wikimedia.org/T439429 (10RobH) 03NEW [14:51:22] 06SRE, 06ServiceOps: 8 wikikube in codfw imlementation tracking - https://phabricator.wikimedia.org/T439430 (10RobH) 03NEW [14:51:50] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps: Q3:rack/setup/install 8 new wikikube hosts - https://phabricator.wikimedia.org/T439428#12369881 (10RobH) [14:53:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:53:07] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: 8 wikikube in codfw imlementation tracking - https://phabricator.wikimedia.org/T439430#12369979 (10Clement_Goubert) 05Open→03Stalled p:05Triage→03Medium [14:53:22] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: 26 wikikube in eqiad imlementation tracking - https://phabricator.wikimedia.org/T439429#12369985 (10Clement_Goubert) 05Open→03Stalled p:05Triage→03Medium [14:53:46] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Q3:rack/setup/install 8 new wikikube hosts - https://phabricator.wikimedia.org/T439428#12369991 (10Clement_Goubert) [14:54:31] !log lucaswerkmeister-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344771|feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332)]], [[gerrit:1344772|feat(AccountSetup), styles: Apply correct background and button styles (T436332)]] (duration: 33m 44s) [14:54:35] T436332: Recommended reading exp: Final design review and adjustments for recommended articles - https://phabricator.wikimedia.org/T436332 [14:54:41] 10ops-eqiad, 06SRE, 06DC-Ops, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Q3:rack/setup/install 26 wikikube hosts - https://phabricator.wikimedia.org/T439427#12369998 (10Clement_Goubert) [14:55:12] 06SRE, 10conftool, 06Infrastructure-Foundations: confd fails to start after a reimage - https://phabricator.wikimedia.org/T244477#12370001 (10LSobanski) 05Open→03Resolved a:03LSobanski Unlikely that this is still a problem given the ubiquity of confd. [14:55:32] 06SRE, 06Infrastructure-Foundations: Make notrack available for nftables - https://phabricator.wikimedia.org/T348735#12370006 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff This has been implemented and we have plenty of firewall::service defintions on nftables-enabled systems. [14:55:32] !log UTC afternoon backport+config window done [14:55:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:06:26] !log filippo@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm [15:06:42] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12370089 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1004 for host cloudcephosd1055.eqiad.wmnet with OS bookworm ex... [15:07:37] 06SRE, 06Data-Persistence, 10Data-Persistence-Design-Review, 10Wikifeeds, and 2 others: Provision Cassandra keyspace for WikiFeeds cache - https://phabricator.wikimedia.org/T435450#12370093 (10Eevans) Hi @OSleger-WMF, The process we use for this is https://wikitech.wikimedia.org/wiki/SRE/Data_Persistence/... [15:09:45] (03PS1) 10Santiago Faci: Deploy GrowthBook 5.1.0 to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345571 (https://phabricator.wikimedia.org/T438954) [15:12:01] (03CR) 10Vgutierrez: [C:03+1] Move the pki service entry to service_setup [puppet] - 10https://gerrit.wikimedia.org/r/1338937 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [15:16:27] (03PS4) 10Ebernhardson: semantic ssd test: Drop to 3 masters, fix affinity [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344715 (https://phabricator.wikimedia.org/T438058) [15:17:04] 10ops-eqiad, 06SRE, 06DC-Ops: hw troubleshooting: disk errors for cephosd1002.eqiad.wmnet /c0/e23/s4 - https://phabricator.wikimedia.org/T438952#12370133 (10RobH) >>! In T438952#12357353, @wiki_willy wrote: > @RobH - can you request quote for the disk? Since we're in the middle of figuring out whether we sh... [15:17:13] 10ops-eqiad, 06SRE, 06DC-Ops: hw troubleshooting: disk errors for cephosd1002.eqiad.wmnet /c0/e23/s4 - https://phabricator.wikimedia.org/T438952#12370135 (10RobH) [15:25:20] PROBLEM - Host titan1002 is DOWN: PING CRITICAL - Packet loss = 60%, RTA = 4036.79 ms [15:25:28] RECOVERY - Host titan1002 is UP: PING OK - Packet loss = 0%, RTA = 148.65 ms [15:30:05] jan_drewniak: May I have your attention please! Wikimedia Portals Update. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260928T1530) [15:31:57] (03PS1) 10Ameisenigel: Enable Suggested Investigations on Serbian Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345577 (https://phabricator.wikimedia.org/T439312) [15:32:08] going to be doing a portals deploy shortly [15:32:35] (03CR) 10Dreamy Jazz: [C:04-2] "Database tables need to be created first and also needs PSI signoff" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345577 (https://phabricator.wikimedia.org/T439312) (owner: 10Ameisenigel) [15:34:12] (03PS1) 10CWilliams: versitygw: require mount points for objectstorage [puppet] - 10https://gerrit.wikimedia.org/r/1345578 (https://phabricator.wikimedia.org/T439424) [15:35:57] (03CR) 10Brouberol: [C:03+1] Deploy GrowthBook 5.1.0 to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345571 (https://phabricator.wikimedia.org/T438954) (owner: 10Santiago Faci) [15:37:32] (03PS1) 10Jdrewniak: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345580 (https://phabricator.wikimedia.org/T128546) [15:37:37] (03CR) 10Tryvix1509: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345577 (https://phabricator.wikimedia.org/T439312) (owner: 10Ameisenigel) [15:40:20] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdrewniak@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345580 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [15:40:34] !log elukey@puppetserver1001 conftool action : set/pooled=true; selector: dnsdisc=pki,name=codfw [15:41:30] (03Merged) 10jenkins-bot: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345580 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [15:41:42] !log jdrewniak@deploy1003 Started scap sync-world: Backport for [[gerrit:1345580|Bumping portals to master (T128546)]] [15:41:45] T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics - https://phabricator.wikimedia.org/T128546 [15:41:52] !log elukey@puppetserver1001 conftool action : set/pooled=true; selector: dnsdisc=puppetdb-api,name=codfw [15:41:58] !log elukey@puppetserver1001 conftool action : set/pooled=true; selector: dnsdisc=puppetboard,name=codfw [15:44:57] (03PS1) 10Jgiannelos: prv: Enable parsoid rendering for 4 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345584 (https://phabricator.wikimedia.org/T439444) [15:45:42] !log jdrewniak@deploy1003 jdrewniak: Backport for [[gerrit:1345580|Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:46:52] !log clone wbc_entity_usage to x1 for testwikidatawiki clients # T439319 [15:46:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:46:56] T439319: Move wbc_entity_usage to x1 for testwikidatawiki clients - https://phabricator.wikimedia.org/T439319 [15:56:16] !log jdrewniak@deploy1003 jdrewniak: Continuing with deployment [15:57:24] (03CR) 10Ozge: ml-services: expose gpt-oss-safeguard-20b publicly (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345562 (https://phabricator.wikimedia.org/T439395) (owner: 10Ilias Sarantopoulos) [15:59:07] (03PS6) 10JHathaway: lvm: switch to upstream repo [puppet] - 10https://gerrit.wikimedia.org/r/1345334 [15:59:10] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345334 (owner: 10JHathaway) [16:00:17] (03CR) 10Santiago Faci: [C:03+2] Deploy GrowthBook 5.1.0 to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345571 (https://phabricator.wikimedia.org/T438954) (owner: 10Santiago Faci) [16:01:36] !log jdrewniak@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345580|Bumping portals to master (T128546)]] (duration: 19m 54s) [16:01:39] T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics - https://phabricator.wikimedia.org/T128546 [16:01:54] (03Merged) 10jenkins-bot: Deploy GrowthBook 5.1.0 to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345571 (https://phabricator.wikimedia.org/T438954) (owner: 10Santiago Faci) [16:02:46] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply [16:03:36] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply [16:04:48] jouncebot: nowandnext [16:04:48] No deployments scheduled for the next 0 hour(s) and 55 minute(s) [16:04:48] In 0 hour(s) and 55 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260928T1700) [16:04:48] In 0 hour(s) and 55 minute(s): Wikidata Query Service weekly deploy (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260928T1700) [16:04:56] !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_drmrs - 7.1.1-2~bpo13+wmf3 () [16:05:11] !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_drmrs - 7.1.1-2~bpo13+wmf3 () [16:05:18] (03CR) 10Zabe: [C:03+2] wikibase: Set virtual-wikibase-entityusage to x1 for test wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345298 (https://phabricator.wikimedia.org/T439319) (owner: 10Zabe) [16:06:37] (03Merged) 10jenkins-bot: wikibase: Set virtual-wikibase-entityusage to x1 for test wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345298 (https://phabricator.wikimedia.org/T439319) (owner: 10Zabe) [16:06:52] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1345298|wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] [16:06:56] T439319: Move wbc_entity_usage to x1 for testwikidatawiki clients - https://phabricator.wikimedia.org/T439319 [16:07:46] (03PS1) 10Aude: Enable ReadingLists on all Wikipedia wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345588 (https://phabricator.wikimedia.org/T434924) [16:08:29] (03CR) 10JHathaway: "ready for review" [puppet] - 10https://gerrit.wikimedia.org/r/1345334 (owner: 10JHathaway) [16:10:45] !log zabe@deploy1003 zabe: Backport for [[gerrit:1345298|wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:14:29] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 28 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345588 (https://phabricator.wikimedia.org/T434924) (owner: 10Aude) [16:15:27] !log zabe@deploy1003 zabe: Continuing with deployment [16:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:20:42] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345298|wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)]] (duration: 13m 50s) [16:20:46] T439319: Move wbc_entity_usage to x1 for testwikidatawiki clients - https://phabricator.wikimedia.org/T439319 [16:21:16] (03PS1) 10Herron: sirenbot: persist sre managers rotation name [puppet] - 10https://gerrit.wikimedia.org/r/1345591 (https://phabricator.wikimedia.org/T438657) [16:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:27:50] 06SRE, 06SRE Observability, 13Patch-For-Review: Chanserv Feature Request - Add the manager oncall in the topic of -sre-private and -operations - https://phabricator.wikimedia.org/T438657#12370467 (10herron) 05Open→03Resolved Manager on call is now included in channel topics! ` @ChanServ set the topi... [16:30:04] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T439299#12370480 (10Jclark-ctr) [16:37:26] (03CR) 10Andrea Denisse: [C:03+1] "Toppisimo!!" [puppet] - 10https://gerrit.wikimedia.org/r/1345569 (https://phabricator.wikimedia.org/T438454) (owner: 10Tiziano Fogli) [16:40:24] 07sre-alert-triage, 06Infrastructure-Foundations, 10Kafka-Infrastructure, 06ServiceOps, and 2 others: Alert in need of triage: PKICertificateExpiry (instance pki2002:9100) - https://phabricator.wikimedia.org/T438703#12370533 (10JMeybohm) [16:50:52] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: db2226 PSU loss of redundancy - https://phabricator.wikimedia.org/T439205#12370557 (10Jhancock.wm) no, it's hot swappable. ty! [16:59:13] (03CR) 10Volans: [V:03+2 C:03+2] cumin: Add dummy insetup role report config [labs/private] - 10https://gerrit.wikimedia.org/r/1345507 (owner: 10Volans) [16:59:19] (03CR) 10Volans: [C:03+2] profile::cumin: Move insetup config to hiera [puppet] - 10https://gerrit.wikimedia.org/r/1345508 (owner: 10Volans) [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260928T1700) [17:00:05] ryankemper: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for Wikidata Query Service weekly deploy . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260928T1700). [17:01:27] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:03:32] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 214718912 and 17 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [17:05:09] (03CR) 10Andrew Bogott: [C:03+2] "ack" [puppet] - 10https://gerrit.wikimedia.org/r/1191027 (https://phabricator.wikimedia.org/T404249) (owner: 10Andrew Bogott) [17:06:04] PROBLEM - Host mr1-eqsin.oob is DOWN: PING CRITICAL - Packet loss = 100% [17:06:32] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 129624 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [17:10:46] PROBLEM - Confd vcl based reload on cp6015 is CRITICAL: reload-vcl failed to run since 0h, 2 minutes. https://wikitech.wikimedia.org/wiki/Varnish [17:10:46] PROBLEM - Confd vcl based reload on cp6014 is CRITICAL: reload-vcl failed to run since 0h, 2 minutes. https://wikitech.wikimedia.org/wiki/Varnish [17:10:46] PROBLEM - Confd vcl based reload on cp6012 is CRITICAL: reload-vcl failed to run since 0h, 2 minutes. https://wikitech.wikimedia.org/wiki/Varnish [17:11:06] RECOVERY - Host mr1-eqsin.oob is UP: PING OK - Packet loss = 0%, RTA = 218.72 ms [17:15:46] RECOVERY - Confd vcl based reload on cp6015 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [17:15:46] RECOVERY - Confd vcl based reload on cp6014 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [17:22:53] (03PS6) 10Andrew Bogott: test_cookbook.py: Allow recording tests on invoked cookbooks [puppet] - 10https://gerrit.wikimedia.org/r/1290858 [17:24:49] (03CR) 10Andrew Bogott: [C:03+2] cloudnfs: Remove fastcci NFS configuration [puppet] - 10https://gerrit.wikimedia.org/r/1325913 (owner: 10Majavah) [17:27:23] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [17:28:20] (03PS1) 10Giuseppe Lavagetto: kubernetes::deployment_server: setup daily timer to reload certificates [puppet] - 10https://gerrit.wikimedia.org/r/1345600 (https://phabricator.wikimedia.org/T439409) [17:40:46] RECOVERY - Confd vcl based reload on cp6012 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [17:47:55] (03PS1) 10Subramanya Sastry: Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid" [core] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345604 [17:48:35] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 28 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [core] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345604 (owner: 10Subramanya Sastry) [17:48:52] !log dzahn@cumin1004 START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie [17:49:12] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12370790 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1004 for host zuul1006.eqiad.wmnet with OS tr... [17:51:29] (03PS1) 101Veertje: Move Gerrit login styling into GerritSite.css [puppet] - 10https://gerrit.wikimedia.org/r/1345606 [17:53:34] (03PS3) 10RLazarus: alertmanager: Repoint serviceops-task to the correct project [puppet] - 10https://gerrit.wikimedia.org/r/1344095 [17:53:51] (03PS1) 10DLynch: BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set [extensions/VisualEditor] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345608 (https://phabricator.wikimedia.org/T400506) [17:53:56] (03CR) 10Andrew Bogott: [C:03+1] profile::zookeeper::firewall: Also allow passing a list of hosts [puppet] - 10https://gerrit.wikimedia.org/r/1272766 (owner: 10Muehlenhoff) [17:54:26] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 28 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [extensions/VisualEditor] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345608 (https://phabricator.wikimedia.org/T400506) (owner: 10DLynch) [17:55:11] (03CR) 10RLazarus: [C:03+2] alertmanager: Repoint serviceops-task to the correct project [puppet] - 10https://gerrit.wikimedia.org/r/1344095 (owner: 10RLazarus) [17:56:26] (03PS5) 101Veertje: Fix login form: responsive labels and mobile-friendly layout [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 [17:56:28] (03PS3) 101Veertje: Add daily background rotation with attribution and credit fixes [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1306065 [17:57:14] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1260004 (https://phabricator.wikimedia.org/T420921) (owner: 10Majavah) [18:00:38] (03PS1) 10Lizimedia: Add lizf to list of ops-limited users [puppet] - 10https://gerrit.wikimedia.org/r/1345610 [18:04:23] !log dzahn@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage [18:06:17] (03CR) 10BCornwall: [C:03+1] wmnet: move codfw, eqsin, ulsfo etcd (RO) client SRV records to codfw [dns] - 10https://gerrit.wikimedia.org/r/1344391 (owner: 10Scott French) [18:07:37] (03CR) 10BCornwall: "Rebase as well as replacement of the placeholders." [alerts] - 10https://gerrit.wikimedia.org/r/1217262 (owner: 10CDobbins) [18:08:35] !log dzahn@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage [18:12:50] (03CR) 10Bking: [C:03+1] Define the postgresql-growthbook-fundraising discovery record [dns] - 10https://gerrit.wikimedia.org/r/1345543 (https://phabricator.wikimedia.org/T438350) (owner: 10Brouberol) [18:13:18] (03CR) 10Bking: [C:03+1] deployment_server: define the postgresql-growthbook-fundraising kubeconfigs [puppet] - 10https://gerrit.wikimedia.org/r/1345540 (https://phabricator.wikimedia.org/T438350) (owner: 10Brouberol) [18:19:56] !log cdobbins@cumin1004 START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie [18:24:22] !log dzahn@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1006.eqiad.wmnet with OS trixie [18:24:37] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12370896 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1004 for host zuul1006.eqiad.wmnet with OS trixie... [18:25:56] (03CR) 10JHathaway: [C:03+2] Add lizf to list of ops-limited users [puppet] - 10https://gerrit.wikimedia.org/r/1345610 (owner: 10Lizimedia) [18:27:11] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [18:30:38] (03CR) 10Btullis: [C:03+1] "Looks good to me." [puppet] - 10https://gerrit.wikimedia.org/r/1345334 (owner: 10JHathaway) [18:31:26] (03CR) 10JHathaway: [C:03+2] lvm: switch to upstream repo [puppet] - 10https://gerrit.wikimedia.org/r/1345334 (owner: 10JHathaway) [18:31:59] rzl: can I merge in your alert manager patch? [18:32:24] jhathaway: oh, please do [18:32:30] o7 [18:32:34] thanks! [18:33:26] I thought I'd already done that. apparently I ran "run-puppet-agent" instead of "puppet-merge" and then incredibly I didn't notice the wrong thing happened, when it finished I just closed the tab and moved on [18:33:41] lol [18:33:45] sounds like my life [18:36:11] FIRING: [4x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:36:30] !log dancy@deploy1003 Installing scap version "4.293.0" for 3 host(s) [18:38:25] !log dancy@deploy1003 Installation of scap version "4.293.0" completed for 3 hosts [18:39:25] (03PS1) 10VolkerE: Enable Reading Recommendations experiment featured category setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345619 (https://phabricator.wikimedia.org/T437348) [18:41:38] (03CR) 10Andrew Bogott: "you're right! Fixed." [puppet] - 10https://gerrit.wikimedia.org/r/1290858 (owner: 10Andrew Bogott) [18:42:18] (03CR) 10Andrew Bogott: [C:03+2] cloud-vps vendordata: install cumin key at VM creation time (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1302236 (https://phabricator.wikimedia.org/T422801) (owner: 10Andrew Bogott) [18:42:36] (03CR) 10Andrew Bogott: [C:03+2] Add upstream repos for openstack flamingo and gazpacho (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1276009 (https://phabricator.wikimedia.org/T423598) (owner: 10Andrew Bogott) [18:42:58] (03CR) 10Andrew Bogott: [C:03+2] Neutron policy: Update port policies (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1163003 (owner: 10Andrew Bogott) [18:45:41] (03Abandoned) 10Andrew Bogott: deployment-prep: remove unused SAN list [puppet] - 10https://gerrit.wikimedia.org/r/511335 (owner: 10Alex Monk) [18:45:46] PROBLEM - Confd vcl based reload on cp6014 is CRITICAL: reload-vcl failed to run since 0h, 2 minutes. https://wikitech.wikimedia.org/wiki/Varnish [18:46:38] !log puppet lvm upgrade broke ml-serve hosts, looking for a hotfix, or I'll rollback... [18:46:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:47:29] !log brett@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_drmrs - 7.1.1-2~bpo13+wmf3 () [18:47:33] (03CR) 10Bking: [C:03+1] dse-k8s-eqiad: define the postgresql-growthbook-fundraising ns [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345537 (https://phabricator.wikimedia.org/T438350) (owner: 10Brouberol) [18:47:40] (03CR) 10Bking: [C:03+1] dse-k8s-eqiad: define the postgresql-growthbook-fundraising release [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345538 (https://phabricator.wikimedia.org/T438350) (owner: 10Brouberol) [18:48:43] (03CR) 10Aude: [C:04-1] Enable Reading Recommendations experiment featured category setting (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345619 (https://phabricator.wikimedia.org/T437348) (owner: 10VolkerE) [18:51:46] RECOVERY - Confd vcl based reload on cp6014 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [19:03:03] !log cdobbins@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie [19:03:25] !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 () [19:05:56] (03PS1) 10Lizimedia: i added the wrong keys :( [puppet] - 10https://gerrit.wikimedia.org/r/1345623 [19:06:46] (03CR) 10JHathaway: [C:03+2] i added the wrong keys :( [puppet] - 10https://gerrit.wikimedia.org/r/1345623 (owner: 10Lizimedia) [19:06:48] (03CR) 10JHathaway: [V:03+2 C:03+2] i added the wrong keys :( [puppet] - 10https://gerrit.wikimedia.org/r/1345623 (owner: 10Lizimedia) [19:12:12] (03PS1) 10VolkerE: fix(ReadingRecommendations): reposition personalize button [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345624 (https://phabricator.wikimedia.org/T439158) [19:12:28] (03CR) 10VolkerE: [C:03+1] fix(ReadingRecommendations): reposition personalize button [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345624 (https://phabricator.wikimedia.org/T439158) (owner: 10VolkerE) [19:13:55] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 28 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345624 (https://phabricator.wikimedia.org/T439158) (owner: 10VolkerE) [19:15:17] (03CR) 10VolkerE: Enable Reading Recommendations experiment featured category setting (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345619 (https://phabricator.wikimedia.org/T437348) (owner: 10VolkerE) [19:15:52] (03PS1) 10Matthias Mullie: Skip expensive work if known to be useless [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345626 (https://phabricator.wikimedia.org/T439182) [19:16:33] (03PS2) 101Veertje: Move Gerrit login styling and background rotation into GerritSite.css [puppet] - 10https://gerrit.wikimedia.org/r/1345606 [19:16:39] !log brett@cumin1004 END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_drmrs - 7.1.1-2~bpo13+wmf3 () [19:16:53] (03PS6) 101Veertje: Fix login form: responsive labels and mobile-friendly layout [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 [19:16:55] (03PS4) 101Veertje: Add daily background rotation with attribution and credit fixes [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1306065 [19:16:56] (03PS1) 10Matthias Mullie: Make carousel images are visible in the margins [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345627 (https://phabricator.wikimedia.org/T439422) [19:24:24] (03CR) 10Aude: [C:04-1] Enable Reading Recommendations experiment featured category setting (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345619 (https://phabricator.wikimedia.org/T437348) (owner: 10VolkerE) [19:24:26] (03PS1) 10Matthias Mullie: carousel: fix carousel image preload observer root [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345633 (https://phabricator.wikimedia.org/T439422) [19:24:26] (03PS3) 101Veertje: Move Gerrit login styling and background rotation into GerritSite.css [puppet] - 10https://gerrit.wikimedia.org/r/1345606 [19:24:46] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 29 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345626 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [19:25:26] (03PS1) 10Matthias Mullie: Cache extracted carousel data [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345632 (https://phabricator.wikimedia.org/T439182) [19:25:42] FIRING: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [19:25:44] (03CR) 10CI reject: [V:04-1] Cache extracted carousel data [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345632 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [19:25:46] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 29 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345632 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [19:25:53] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 29 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345627 (https://phabricator.wikimedia.org/T439422) (owner: 10Matthias Mullie) [19:26:01] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 29 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345633 (https://phabricator.wikimedia.org/T439422) (owner: 10Matthias Mullie) [19:27:05] (03CR) 10Matthias Mullie: "This one follows https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MultimediaViewer/+/1345626 - not sure how to properly stack it thro" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345632 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [19:27:12] (03PS2) 10VolkerE: Enable Reading Recommendations experiment featured category setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345619 (https://phabricator.wikimedia.org/T437348) [19:27:30] (03CR) 10VolkerE: Enable Reading Recommendations experiment featured category setting (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345619 (https://phabricator.wikimedia.org/T437348) (owner: 10VolkerE) [19:28:25] (03CR) 10Jdlrobson: [C:03+1] Enable ReadingLists on all Wikipedia wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345588 (https://phabricator.wikimedia.org/T434924) (owner: 10Aude) [19:29:03] (03PS4) 101Veertje: Move Gerrit login styling into GerritSite.css [puppet] - 10https://gerrit.wikimedia.org/r/1345606 [19:29:06] (03PS1) 101Veertje: Add rotating Commons background to the Gerrit login page [puppet] - 10https://gerrit.wikimedia.org/r/1345634 [19:30:54] (03CR) 10VolkerE: Enable ReadingLists on all Wikipedia wikis (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345588 (https://phabricator.wikimedia.org/T434924) (owner: 10Aude) [19:30:59] (03CR) 10VolkerE: [C:03+1] Enable ReadingLists on all Wikipedia wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345588 (https://phabricator.wikimedia.org/T434924) (owner: 10Aude) [19:31:02] (03Abandoned) 101Veertje: Add daily background rotation with attribution and credit fixes [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1306065 (owner: 101Veertje) [19:31:27] !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp601[3-6].drmrs.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 () [19:32:33] (03CR) 10Aude: Enable ReadingLists on all Wikipedia wikis (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345588 (https://phabricator.wikimedia.org/T434924) (owner: 10Aude) [19:34:40] (03CR) 10Aude: [C:03+1] Enable Reading Recommendations experiment featured category setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345619 (https://phabricator.wikimedia.org/T437348) (owner: 10VolkerE) [19:35:08] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 28 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345619 (https://phabricator.wikimedia.org/T437348) (owner: 10VolkerE) [19:35:25] (03PS2) 10Aude: Enable ReadingLists on all Wikipedia wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345588 (https://phabricator.wikimedia.org/T434924) [19:35:42] RESOLVED: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [19:38:38] (03PS1) 10DLynch: EditCheck: only log a session if checks are actually possible [extensions/VisualEditor] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345637 (https://phabricator.wikimedia.org/T438916) [19:38:51] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 28 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [extensions/VisualEditor] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345637 (https://phabricator.wikimedia.org/T438916) (owner: 10DLynch) [19:40:43] (03PS2) 101Veertje: Add rotating Commons background to the Gerrit login page [puppet] - 10https://gerrit.wikimedia.org/r/1345634 [19:42:10] !log dancy@deploy1003 Installing scap version "4.293.1" for 3 host(s) [19:42:38] (03PS5) 10JHathaway: stdlib: upgrade to v10.1.0 [puppet] - 10https://gerrit.wikimedia.org/r/1345157 (https://phabricator.wikimedia.org/T438912) [19:44:01] (03PS3) 101Veertje: Add rotating Commons background to the Gerrit login page [puppet] - 10https://gerrit.wikimedia.org/r/1345634 [19:44:03] !log dancy@deploy1003 Installation of scap version "4.293.1" completed for 3 hosts [19:44:25] (03PS3) 10SBassett: Content Security Policy: add stricter report-only policy [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344774 (https://phabricator.wikimedia.org/T419612) [19:45:04] (03PS4) 101Veertje: Add rotating Commons background to the Gerrit login page [puppet] - 10https://gerrit.wikimedia.org/r/1345634 [19:46:35] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345157 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [19:48:05] (03PS1) 10VolkerE: fix(Homepage): apply the body transform only when an overlay is open [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345643 (https://phabricator.wikimedia.org/T439038) [19:48:28] (03CR) 10VolkerE: [C:03+1] fix(Homepage): apply the body transform only when an overlay is open [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345643 (https://phabricator.wikimedia.org/T439038) (owner: 10VolkerE) [19:48:50] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, September 28 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345643 (https://phabricator.wikimedia.org/T439038) (owner: 10VolkerE) [19:49:56] (03PS5) 101Veertje: Add rotating Commons background to the Gerrit login page [puppet] - 10https://gerrit.wikimedia.org/r/1345634 [19:50:03] (03CR) 10CDobbins: [C:03+2] prometheus: fix prometheus-ferm-mss.py [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) (owner: 10CDobbins) [19:50:44] (03PS1) 10DLynch: mesh: Copy service 1.2.1 to 1.2.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345645 (https://phabricator.wikimedia.org/T436689) [19:50:46] (03PS4) 10SBassett: Content Security Policy: add stricter report-only policy [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344774 (https://phabricator.wikimedia.org/T419612) [19:50:46] (03PS1) 10SBassett: Remove foundationwiki's CSP report-only directive [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345649 (https://phabricator.wikimedia.org/T423691) [19:50:47] (03PS1) 10DLynch: mesh.service: Add public_port_app_protocol [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345646 (https://phabricator.wikimedia.org/T436689) [19:50:49] (03PS1) 10DLynch: python-webapp: Update mesh.service to 1.2.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345647 (https://phabricator.wikimedia.org/T436689) [19:50:52] (03PS1) 10DLynch: editcheck-headless: Declare the TLS port as gRPC [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345648 (https://phabricator.wikimedia.org/T436689) [19:51:06] (03CR) 10JHathaway: [C:03+2] "definitely, but I missed this comment before merging, I was a bit trigger happy, sorry" [puppet] - 10https://gerrit.wikimedia.org/r/1345334 (owner: 10JHathaway) [19:51:45] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 29 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345649 (https://phabricator.wikimedia.org/T423691) (owner: 10SBassett) [19:52:40] (03CR) 10SBassett: [C:04-1] "Hold for config deploy." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345649 (https://phabricator.wikimedia.org/T423691) (owner: 10SBassett) [19:54:40] !log dzahn@cumin1004 START - Cookbook sre.hosts.reimage for host zuul1007.eqiad.wmnet with OS trixie [19:54:52] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12371323 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1004 for host zuul1007.eqiad.wmnet with OS tr... [19:55:36] (03Abandoned) 10DLynch: ingress: Copy istio 1.4.0 to 1.4.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344833 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:55:43] (03Abandoned) 10DLynch: ingress.istio: Add useClientProtocol [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344834 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:55:51] (03Abandoned) 10DLynch: python-webapp: Update ingress.istio to 1.4.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344835 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:56:02] (03Abandoned) 10DLynch: editcheck-headless: Forward gRPC as HTTP/2 through the ingress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344803 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:56:34] (03Abandoned) 10DLynch: mesh: Copy service 1.1.0 to 1.1.1 and configuration 1.15.3 to 1.15.4 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344804 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:56:42] (03Abandoned) 10DLynch: mesh: Add alpn_protocols and public_port_app_protocol [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344805 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:56:49] (03Abandoned) 10DLynch: python-webapp: Update mesh.configuration to 1.15.4 and service to 1.1.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344806 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:56:58] (03Abandoned) 10DLynch: editcheck-headless: Use HTTP/2 through the mesh for gRPC [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344807 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: I, the Bot under the Fountain, call upon thee, The Deployer, to do UTC late backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260928T2000). [20:00:05] aude, subbu, kemayo, and Volker_E: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:25] o/ [20:00:26] I can deploy the config patches [20:00:41] I don't have config patches, but don't mind doing my own deploy for them. [20:00:47] o/ [20:00:48] I will proceed with them [20:01:06] (03CR) 10TrainBranchBot: [C:03+2] "Approved by aude@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345588 (https://phabricator.wikimedia.org/T434924) (owner: 10Aude) [20:01:07] (03CR) 10TrainBranchBot: [C:03+2] "Approved by aude@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345619 (https://phabricator.wikimedia.org/T437348) (owner: 10VolkerE) [20:01:29] i'll take care of Volker's patches and if anyone else's in need of a deployer [20:01:44] can someone else deploy mine? [20:02:03] sure [20:02:05] ty [20:02:40] (03Merged) 10jenkins-bot: Enable ReadingLists on all Wikipedia wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345588 (https://phabricator.wikimedia.org/T434924) (owner: 10Aude) [20:02:42] (03Merged) 10jenkins-bot: Enable Reading Recommendations experiment featured category setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345619 (https://phabricator.wikimedia.org/T437348) (owner: 10VolkerE) [20:03:01] !log aude@deploy1003 Started scap sync-world: Backport for [[gerrit:1345588|Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619|Enable Reading Recommendations experiment featured category setting (T437348)]] [20:03:07] T434924: Enable Reading Lists for all logged-in users [Sept 28] on all remaining Wikipedia wikis - https://phabricator.wikimedia.org/T434924 [20:03:07] T437348: Recommended reading exp: Bug bash - https://phabricator.wikimedia.org/T437348 [20:05:30] (03PS5) 101Veertje: Move Gerrit login styling into GerritSite.css [puppet] - 10https://gerrit.wikimedia.org/r/1345606 [20:05:57] (03PS6) 101Veertje: Move Gerrit login styling into GerritSite.css [puppet] - 10https://gerrit.wikimedia.org/r/1345606 [20:06:00] (03PS6) 101Veertje: Add rotating Commons background to the Gerrit login page [puppet] - 10https://gerrit.wikimedia.org/r/1345634 [20:07:08] !log aude@deploy1003 aude, volker-e: Backport for [[gerrit:1345588|Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619|Enable Reading Recommendations experiment featured category setting (T437348)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:08:29] !log brett@puppetserver1001 conftool action : set/pooled=yes; selector: name=cp6013.* [20:08:36] !log dzahn@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage [20:08:52] i am verifying things [20:10:45] !log aude@deploy1003 aude, volker-e: Continuing with deployment [20:13:03] !log dzahn@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage [20:16:07] !log aude@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345588|Enable ReadingLists on all Wikipedia wikis (T434924)]], [[gerrit:1345619|Enable Reading Recommendations experiment featured category setting (T437348)]] (duration: 13m 05s) [20:16:11] T434924: Enable Reading Lists for all logged-in users [Sept 28] on all remaining Wikipedia wikis - https://phabricator.wikimedia.org/T434924 [20:16:12] T437348: Recommended reading exp: Bug bash - https://phabricator.wikimedia.org/T437348 [20:16:33] I am done [20:17:30] i'll do the next one and then pass over to Kemayo [20:18:37] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cjming@deploy1003 using scap backport" [core] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345604 (owner: 10Subramanya Sastry) [20:22:11] (03Merged) 10jenkins-bot: Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid" [core] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345604 (owner: 10Subramanya Sastry) [20:22:25] !log cjming@deploy1003 Started scap sync-world: Backport for [[gerrit:1345604|Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] [20:26:24] !log cjming@deploy1003 ssastry, cjming: Backport for [[gerrit:1345604|Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:26:59] !log cjming@deploy1003 ssastry, cjming: Continuing with deployment [20:27:02] ok .. will test. [20:27:21] oh sorry - i went ahead and sync'd [20:27:58] lgtm. it works. :) [20:28:16] oh good [20:29:20] !log dzahn@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1007.eqiad.wmnet with OS trixie [20:29:28] Kemayo, in case anyone complains about dirty diffs via VE in the timeframe between Thu and Monday ... that patch fixes that. https://phabricator.wikimedia.org/T439453 has info. FYI. [20:29:33] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12371668 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1004 for host zuul1007.eqiad.wmnet with OS trixie... [20:32:12] !log cjming@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345604|Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"]] (duration: 09m 47s) [20:32:18] Kemayo: over to you - lmk when you're done [20:32:29] cjming: Thanks, will do. [20:32:57] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kemayo@deploy1003 using scap backport" [extensions/VisualEditor] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345608 (https://phabricator.wikimedia.org/T400506) (owner: 10DLynch) [20:32:57] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kemayo@deploy1003 using scap backport" [extensions/VisualEditor] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345637 (https://phabricator.wikimedia.org/T438916) (owner: 10DLynch) [20:35:09] 06SRE, 06Traffic: Prometheus Ferm MSS export does not work for IPv6 - https://phabricator.wikimedia.org/T433672#12371697 (10CDobbins) @cmooney, this should be fixed now. As of 20:20 UTC today, /var/lib/prometheus/node.d/ferm-mss.prom's contents on clouddumps1002 were: ` # HELP ferm_mss_cfg MSS values for Ferm-... [20:37:47] PROBLEM - Confd vcl based reload on cp6016 is CRITICAL: reload-vcl failed to run since 0h, 2 minutes. https://wikitech.wikimedia.org/wiki/Varnish [20:41:02] (03Merged) 10jenkins-bot: BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set [extensions/VisualEditor] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345608 (https://phabricator.wikimedia.org/T400506) (owner: 10DLynch) [20:41:04] (03Merged) 10jenkins-bot: EditCheck: only log a session if checks are actually possible [extensions/VisualEditor] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345637 (https://phabricator.wikimedia.org/T438916) (owner: 10DLynch) [20:41:21] !log kemayo@deploy1003 Started scap sync-world: Backport for [[gerrit:1345608|BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637|EditCheck: only log a session if checks are actually possible (T438916)]] [20:41:26] T400506: Enable volunteers to configure what namespaces Edit Checks have the potential to show within - https://phabricator.wikimedia.org/T400506 [20:41:26] T438916: Make Suggestion Mode usage data publicly available - https://phabricator.wikimedia.org/T438916 [20:45:22] !log kemayo@deploy1003 kemayo: Backport for [[gerrit:1345608|BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637|EditCheck: only log a session if checks are actually possible (T438916)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:46:34] !log kemayo@deploy1003 kemayo: Continuing with deployment [20:47:13] !log dzahn@cumin1004 START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie [20:47:30] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12371765 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1004 for host zuul1005.eqiad.wmnet with OS tr... [20:47:47] !log uploaded rsyslog 8.2608.0-4~wmf13+1 to trixie-wikimedia T438375 [20:47:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:47:50] T438375: rsyslog ratelimit / mediawiki log producers ratelimit - https://phabricator.wikimedia.org/T438375 [20:51:49] !log kemayo@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345608|BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506)]], [[gerrit:1345637|EditCheck: only log a session if checks are actually possible (T438916)]] (duration: 10m 28s) [20:51:52] cjming: Back to you [20:51:54] T400506: Enable volunteers to configure what namespaces Edit Checks have the potential to show within - https://phabricator.wikimedia.org/T400506 [20:51:54] T438916: Make Suggestion Mode usage data publicly available - https://phabricator.wikimedia.org/T438916 [20:52:09] ty [20:53:07] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cjming@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345624 (https://phabricator.wikimedia.org/T439158) (owner: 10VolkerE) [20:53:07] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cjming@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345643 (https://phabricator.wikimedia.org/T439038) (owner: 10VolkerE) [20:57:39] (03Merged) 10jenkins-bot: fix(ReadingRecommendations): reposition personalize button [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345624 (https://phabricator.wikimedia.org/T439158) (owner: 10VolkerE) [21:00:04] alexsanford, Reedy, sbassett, Maryum, and manfredi: Weekly Security deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260928T2100). Please do the needful. [21:00:19] (03Merged) 10jenkins-bot: fix(Homepage): apply the body transform only when an overlay is open [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345643 (https://phabricator.wikimedia.org/T439038) (owner: 10VolkerE) [21:00:37] !log cjming@deploy1003 Started scap sync-world: Backport for [[gerrit:1345624|fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643|fix(Homepage): apply the body transform only when an overlay is open (T439038)]] [21:00:42] T439158: [beta cluster] Mobile: Daily reads header - growthexperiments-reading-recommendations-icon--configure is placed centrally - https://phabricator.wikimedia.org/T439158 [21:00:42] T439038: [testwiki wmf.21] Daily reads - interest selector dialog is not displayed in viewport - https://phabricator.wikimedia.org/T439038 [21:04:52] !log cjming@deploy1003 volker-e, cjming: Backport for [[gerrit:1345624|fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643|fix(Homepage): apply the body transform only when an overlay is open (T439038)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:06:32] i think Volker is testing now [21:06:49] RECOVERY - Confd vcl based reload on cp6016 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [21:07:06] !log arlolra@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [21:07:44] !log arlolra@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [21:07:45] !log arlolra@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [21:08:02] syncing [21:08:07] !log cjming@deploy1003 volker-e, cjming: Continuing with deployment [21:08:30] !log arlolra@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [21:09:23] (03Abandoned) 10Andrea Denisse: WIP [DNM] Add SQLite-backed incident store and analytics CLI [software/klaxon] - 10https://gerrit.wikimedia.org/r/1274026 (https://phabricator.wikimedia.org/T431101) (owner: 10CDanis) [21:12:55] !log brett@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp601[3-6].drmrs.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 () [21:13:24] !log cjming@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345624|fix(ReadingRecommendations): reposition personalize button (T439158)]], [[gerrit:1345643|fix(Homepage): apply the body transform only when an overlay is open (T439038)]] (duration: 12m 47s) [21:13:28] T439158: [beta cluster] Mobile: Daily reads header - growthexperiments-reading-recommendations-icon--configure is placed centrally - https://phabricator.wikimedia.org/T439158 [21:13:28] T439038: [testwiki wmf.21] Daily reads - interest selector dialog is not displayed in viewport - https://phabricator.wikimedia.org/T439038 [21:17:54] (03PS1) 10Herron: upgrade rsyslog to 8.2608.0-4 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1345655 (https://phabricator.wikimedia.org/T438375) [21:20:03] Hey all - have one private sec patch to deploy. Let me know if I should hold off… [21:24:52] !log deleting suspect coredns pods showing upstream resolution health check failures in eqiad - T439201 [21:24:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:24:58] T439201: MediaWiki periodic job update-special-pages-s8 failed - https://phabricator.wikimedia.org/T439201 [21:28:39] (03CR) 10JHathaway: "Ready for review. I spot checked the edits, and they all look correct." [puppet] - 10https://gerrit.wikimedia.org/r/1345157 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [21:29:01] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [21:37:39] dzahn@cumin1004 reimage (PID 1450772) is awaiting input [21:41:50] !log eevans@deploy1003 helmfile [staging] START helmfile.d/services/sessionstore: sync [21:41:53] !log eevans@deploy1003 helmfile [staging] DONE helmfile.d/services/sessionstore: sync [21:42:05] !log eevans@deploy1003 helmfile [eqiad] START helmfile.d/services/sessionstore: sync [21:42:14] !log eevans@deploy1003 helmfile [eqiad] DONE helmfile.d/services/sessionstore: sync [21:42:38] (03PS2) 10Herron: upgrade rsyslog to 8.2608.0-4 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1345655 (https://phabricator.wikimedia.org/T438375) [21:43:24] !log brett@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 () [21:44:45] !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 () [21:47:13] !log Deployed security mitigation for T435455 [21:47:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:50:24] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!!" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1345655 (https://phabricator.wikimedia.org/T438375) (owner: 10Herron) [22:20:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [22:21:49] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [22:22:16] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [22:25:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [22:29:01] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [22:39:01] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:47:35] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 734319720 and 26 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [22:49:35] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 2217824 and 1 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [23:00:04] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260928T2300) [23:03:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [23:08:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [23:28:10] (03PS1) 10Dduvall: WIP: Refactor tests to run during build [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1345668 (https://phabricator.wikimedia.org/T439490) [23:30:09] (03PS2) 10Dduvall: WIP: Refactor tests to run during build [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1345668 (https://phabricator.wikimedia.org/T439490) [23:35:28] !log run script to fix fr_metadata drifts # T420341 [23:35:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:35:31] T420341: PDF file has 0x0 image size in Commons - https://phabricator.wikimedia.org/T420341 [23:38:45] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1345671 [23:38:45] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1345671 (owner: 10TrainBranchBot) [23:44:13] !log dzahn@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie [23:44:25] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12372145 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1004 for host zuul1005.eqiad.wmnet with OS trixie... [23:47:02] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1345671 (owner: 10TrainBranchBot)