[00:07:01] 06Traffic, 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12320556 (10BCornwall) a:03BCornwall [07:21:43] 06Traffic, 06RoadToWiki, 06SRE, 10WMF-General-or-Unknown: Domain "wikipedia.ar" redirects to Arabic instead of Spanish - https://phabricator.wikimedia.org/T437799#12320964 (10CoderAnimeshPathak) [08:17:55] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhautions - https://phabricator.wikimedia.org/T437984 (10ayounsi) 03NEW p:05Triage→03High [08:18:17] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhautions - https://phabricator.wikimedia.org/T437984#12321134 (10ayounsi) [08:28:04] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhautions - https://phabricator.wikimedia.org/T437984#12321156 (10ayounsi) Deployed the workaround on asw1-b3-magru (not showing signs of issues yet) ` ayounsi@asw1-b3-magru> start shell user root Password: root@asw1-b3-magru:RE:0% vty fpc0 Swit... [08:33:23] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12321198 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host cp7010.magru.wmnet with OS trixie [08:47:06] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhautions - https://phabricator.wikimedia.org/T437984#12321264 (10ayounsi) [09:04:37] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhautions - https://phabricator.wikimedia.org/T437984#12321357 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a4925aa6-63a3-4037-92db-c71b5a881bd3) set by ayounsi@cumin1003 for 2:00:00 on 12 host(s) and their services with rea... [09:06:12] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhautions - https://phabricator.wikimedia.org/T437984#12321375 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=c96fe5ee-ddcb-4a50-935b-5314cd5dd078) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their services with reas... [09:28:53] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12321467 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host cp7010.magru.wmnet with OS trixie completed: - cp7010 (**PASS**) - Downtimed on Ici... [09:37:30] FIRING: [6x] LibericaUnhealthyRealserverPooled: Liberica service text-httpslb6_443 has 2 unhealthy realservers pooled on lvs3008:3003 - https://wikitech.wikimedia.org/wiki/Liberica#LibericaUnhealthyRealserverPooled - https://alerts.wikimedia.org/?q=alertname%3DLibericaUnhealthyRealserverPooled [09:42:30] RESOLVED: [6x] LibericaUnhealthyRealserverPooled: Liberica service text-httpslb6_443 has 2 unhealthy realservers pooled on lvs3008:3003 - https://wikitech.wikimedia.org/wiki/Liberica#LibericaUnhealthyRealserverPooled - https://alerts.wikimedia.org/?q=alertname%3DLibericaUnhealthyRealserverPooled [09:42:46] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhautions - https://phabricator.wikimedia.org/T437984#12321554 (10ayounsi) [10:55:30] FIRING: [2x] LibericaDiffFPCheck: Liberica instance lvs7001:9100 control plane status doesn't match with forwarding plane status - https://wikitech.wikimedia.org/wiki/Liberica#LibericaDiffFPCheck - https://alerts.wikimedia.org/?q=alertname%3DLibericaDiffFPCheck [11:34:56] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhautions - https://phabricator.wikimedia.org/T437984#12322288 (10cmooney) FWIW I tried this on //lsw1-a2-codfw// but my session just hangs when I try to enter the FPC vty session: ` cmooney@lsw1-a2-codfw> start shell user root Password: root@lsw... [11:38:37] 10netops, 06Infrastructure-Foundations: Junos file descriptors exhautions - https://phabricator.wikimedia.org/T437984#12322320 (10cmooney) My bad on the above - I'd missed lsw1-a2-codfw was one of the affected devices, so the above shouldn't have been a surprise. [12:30:30] FIRING: [2x] LibericaDiffFPCheck: Liberica instance lvs7001:9100 control plane status doesn't match with forwarding plane status - https://wikitech.wikimedia.org/wiki/Liberica#LibericaDiffFPCheck - https://alerts.wikimedia.org/?q=alertname%3DLibericaDiffFPCheck [12:35:30] RESOLVED: [2x] LibericaDiffFPCheck: Liberica instance lvs7001:9100 control plane status doesn't match with forwarding plane status - https://wikitech.wikimedia.org/wiki/Liberica#LibericaDiffFPCheck - https://alerts.wikimedia.org/?q=alertname%3DLibericaDiffFPCheck [12:35:30] FIRING: [10x] LibericaUnhealthyRealserverPooled: Liberica service gerrit-httpslb6_443 has 1 unhealthy realservers pooled on lvs7001:3003 - https://wikitech.wikimedia.org/wiki/Liberica#LibericaUnhealthyRealserverPooled - https://alerts.wikimedia.org/?q=alertname%3DLibericaUnhealthyRealserverPooled [12:38:00] FIRING: [2x] LibericaDiffFPCheck: Liberica instance lvs7001:9100 control plane status doesn't match with forwarding plane status - https://wikitech.wikimedia.org/wiki/Liberica#LibericaDiffFPCheck - https://alerts.wikimedia.org/?q=alertname%3DLibericaDiffFPCheck [12:40:30] RESOLVED: [10x] LibericaUnhealthyRealserverPooled: Liberica service gerrit-httpslb6_443 has 1 unhealthy realservers pooled on lvs7001:3003 - https://wikitech.wikimedia.org/wiki/Liberica#LibericaUnhealthyRealserverPooled - https://alerts.wikimedia.org/?q=alertname%3DLibericaUnhealthyRealserverPooled [13:11:15] RESOLVED: LibericaDiffFPCheck: Liberica instance lvs7003:9100 control plane status doesn't match with forwarding plane status - https://wikitech.wikimedia.org/wiki/Liberica#LibericaDiffFPCheck - https://grafana.wikimedia.org/d/fa4de97a-7114-48c7-a91a-f56089ef554f/liberica?var-site=magru&var-instance=lvs7003 - https://alerts.wikimedia.org/?q=alertname%3DLibericaDiffFPCheck [14:06:41] 10netops, 06Infrastructure-Foundations, 06SRE: Add link from cloudsw1-e4-eqiad to cloudsw1-f4-eiqad - https://phabricator.wikimedia.org/T372061#12323044 (10cmooney) 05Open→03Declined Closing this one, we will review how to provision the bandwidth when the C8/D5 switches are upgraded. [15:36:36] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12323689 (10RobH) 05Open→03Resolved [15:38:16] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12323711 (10SLyngshede-WMF) [15:47:39] 10netops, 06Traffic, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: Jin onsite work for 2026-09-16 @ 09:00 UTC - https://phabricator.wikimedia.org/T438052 (10RobH) 03NEW p:05Triage→03Medium [15:49:10] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12323817 (10SLyngshede-WMF) [15:53:29] 06Traffic, 06Data-Platform-SRE, 06ServiceOps, 10Wikidata, and 3 others: Migrate the deployment of WDQS UI from wikikube to dse-k8s - https://phabricator.wikimedia.org/T436224#12323864 (10Gehel) p:05Triage→03Medium [15:53:42] 10netops, 06Traffic, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: Jin onsite work for 2026-09-16 @ 09:00 UTC - https://phabricator.wikimedia.org/T438052#12323874 (10RobH) [16:04:31] cdanis: Amir1: just as an FYI, we discussed it in the Traffic meeting today and multiple people in Traffic will be working on moving the upload hosts to text, to speed up the proess [16:04:44] *process. so it's not just going to be Simon [16:08:44] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12323971 (10ssingh) [16:09:32] sukhe: thanks <3 [16:40:03] sukhe: <3 thanks! [16:58:41] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad: replace unmanaged msw in racks A4, B7 & C5 - https://phabricator.wikimedia.org/T438071 (10cmooney) 03NEW p:05Triage→03Low [17:25:44] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324446 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1004 for host cp1101.eqiad.wmnet with OS trixie [17:26:07] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324448 (10BCornwall) [17:26:23] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324449 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2003 for host cp7009.magru.wmnet with OS trixie [17:34:26] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324517 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2003 for host cp4045.ulsfo.wmnet with OS trixie [17:34:29] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324518 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2003 for host cp4046.ulsfo.wmnet with OS trixie [17:37:07] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324530 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1004 for host cp3074.esams.wmnet with OS trixie [17:48:58] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324560 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1003 for host cp2044.codfw.wmnet with OS trixie [17:53:26] 10netops, 06Traffic, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: Jin onsite work for 2026-09-16 @ 09:00 UTC - https://phabricator.wikimedia.org/T438052#12324568 (10RobH) [17:54:39] 10netops, 06Traffic, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: Jin onsite work for 2026-09-16 @ 09:00 UTC - https://phabricator.wikimedia.org/T438052#12324576 (10RobH) [18:01:39] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324593 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2003 for host cp4046.ulsfo.wmnet with OS trixie executed with errors: - cp4046 (**FAIL**) - Downtimed... [18:01:43] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324594 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2003 for host cp4045.ulsfo.wmnet with OS trixie executed with errors: - cp4045 (**FAIL**) - Downtimed... [18:04:06] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324598 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1004 for host cp1101.eqiad.wmnet with OS trixie executed with errors: - cp1101 (**FAIL**) - Downtimed... [18:09:36] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324627 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2003 for host cp4045.ulsfo.wmnet with OS trixie [18:10:24] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324642 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1004 for host cp1101.eqiad.wmnet with OS trixie [18:22:29] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324702 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2003 for host cp7009.magru.wmnet with OS trixie completed: - cp7009 (**PASS**) - Downtimed on Icinga/A... [18:27:09] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324721 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2003 for host cp4045.ulsfo.wmnet with OS trixie executed with errors: - cp4045 (**FAIL**) - Removed fr... [18:31:42] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324742 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1004 for host cp3074.esams.wmnet with OS trixie completed: - cp3074 (**PASS**) - Downtimed on Icinga/A... [18:32:29] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324764 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1003 for host cp2044.codfw.wmnet with OS trixie completed: - cp2044 (**PASS**) - Downtimed on Icing... [18:40:20] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324777 (10ssingh) [18:45:55] 10netops, 06Traffic, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: Jin onsite work for 2026-09-16 @ 09:00 UTC - https://phabricator.wikimedia.org/T438052#12324794 (10ssingh) [18:47:23] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324795 (10CDobbins) [18:50:09] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324800 (10BCornwall) [18:50:54] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324804 (10BCornwall) [18:53:02] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324809 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1004 for host cp1101.eqiad.wmnet with OS trixie completed: - cp1101 (**PASS**) - Removed from Puppet a... [18:55:19] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324817 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2003 for host cp4045.ulsfo.wmnet with OS trixie [18:56:18] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324827 (10ssingh) [19:06:08] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324849 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2003 for host cp4046.ulsfo.wmnet with OS trixie [19:07:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp3075 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=esams&var-instance=cp3075 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [19:10:15] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324860 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2003 for host cp7009.magru.wmnet with OS trixie [19:12:42] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324865 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1004 for host cp3075.esams.wmnet with OS trixie [19:16:40] FIRING: VarnishPrometheusExporterDown: Varnish Exporter on instance cp1103:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [19:17:04] ^ depooled [19:17:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp1103 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=eqiad&var-instance=cp1103 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [19:21:51] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324895 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1004 for host cp1103.eqiad.wmnet with OS trixie [19:26:59] 10netops, 06Traffic, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: Jin onsite work for 2026-09-16 @ 09:00 UTC - https://phabricator.wikimedia.org/T438052#12324913 (10RobH) [19:27:58] 06Traffic, 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12324916 (10Southparkfan) Hi @BCornwall, thank you for your help! I have added the servers to `cache_... [19:37:30] 06Traffic, 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12324962 (10BCornwall) Thanks, @Southparkfan ! I went ahead and moved the float... [19:42:38] brett: shall I drag https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1342025 into the UTC late backport window? [19:43:05] Southparkfan: Up to you! I'm not in any rush [19:43:24] but sure, why not [19:43:40] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324981 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2003 for host cp4045.ulsfo.wmnet with OS trixie completed: - cp4045 (**PASS**) - Removed from Puppet a... [19:44:15] Until this change is deployed, XFF headers of the new cache proxies won't be trusted [19:45:07] It's Beta Cluster, not catastrophic at all. Change seems low-risk though, so could probably get it deployed soon :-) [19:46:44] I have an urgent need to break something, so it's good that XFF is [19:46:51] Ahaha :) [19:47:56] Actually, the leading whitespace at line 17 should be removed [19:48:26] oh shit, good catch [19:48:32] bad copypasta [19:48:38] Not sure if `IPSet` strips leading whitespace in IPv6 addresses, but still.. [19:48:54] done [19:51:07] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324994 (10BCornwall) [19:51:45] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324995 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2003 for host cp4046.ulsfo.wmnet with OS trixie completed: - cp4046 (**PASS**) - Removed from Puppet a... [19:52:21] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12324999 (10BCornwall) [19:53:30] FIRING: LibericaDiffFPCheck: Liberica instance lvs4010:9100 control plane status doesn't match with forwarding plane status - https://wikitech.wikimedia.org/wiki/Liberica#LibericaDiffFPCheck - https://grafana.wikimedia.org/d/fa4de97a-7114-48c7-a91a-f56089ef554f/liberica?var-site=ulsfo&var-instance=lvs4010 - https://alerts.wikimedia.org/?q=alertname%3DLibericaDiffFPCheck [19:58:29] FIRING: [2x] LibericaDiffFPCheck: Liberica instance lvs4008:9100 control plane status doesn't match with forwarding plane status - https://wikitech.wikimedia.org/wiki/Liberica#LibericaDiffFPCheck - https://alerts.wikimedia.org/?q=alertname%3DLibericaDiffFPCheck [20:01:21] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12325052 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2003 for host cp7009.magru.wmnet with OS trixie completed: - cp7009 (**WARN**) - Downtimed on Icinga/A... [20:04:55] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12325069 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host cp2046.codfw.wmnet with OS trixie [20:05:30] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12325075 (10BCornwall) [20:06:45] brett: I think we should do config_relaod on A:liberica-ulsfo [20:06:47] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12325079 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1004 for host cp3075.esams.wmnet with OS trixie completed: - cp3075 (**WARN**) - Downtimed on Icinga/A... [20:06:56] ack, will do [20:07:13] and also add that to the notes. specifically we should check if it resolves this alert [20:08:24] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12325086 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1004 for host cp1103.eqiad.wmnet with OS trixie completed: - cp1103 (**WARN**) - Downtimed on Icinga/A... [20:20:35] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12325115 (10ssingh) [20:23:30] RESOLVED: [2x] LibericaDiffFPCheck: Liberica instance lvs4008:9100 control plane status doesn't match with forwarding plane status - https://wikitech.wikimedia.org/wiki/Liberica#LibericaDiffFPCheck - https://alerts.wikimedia.org/?q=alertname%3DLibericaDiffFPCheck [20:33:26] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12325141 (10BCornwall) [20:41:40] FIRING: VarnishPrometheusExporterDown: Varnish Exporter on instance cp6001:9331 is unreachable - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/000000304/varnish-dc-stats?viewPanel=17 - https://alerts.wikimedia.org/?q=alertname%3DVarnishPrometheusExporterDown [20:41:43] FIRING: HaproxyKafkaExporterDown: HaproxyKafka on cp6001 is down - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaExporterDown - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=drmrs&var-instance=cp6001 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaExporterDown [20:44:04] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12325180 (10BCornwall) @SLyngshede-WMF and @Fabfur: Be advised that we ran into those same annoying PXE boot problems we've had in the past on multiple hosts. We decided to make the [[ https://ge... [20:48:06] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12325190 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host cp2046.codfw.wmnet with OS trixie completed: - cp2046 (**PASS**) - Downtimed on Icing... [20:52:06] brett: your patch is live, congrats :) [20:52:24] Southparkfan: Yay, thanks so much for the help :) [20:53:03] Do you want to proceed with the rest of steps 3 and 4 in https://phabricator.wikimedia.org/T436468#12324916? [20:53:29] yeah, I'll do it [20:53:47] Alright, thanks!! [20:54:11] 06Traffic, 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace cache proxies: deployment-cache-(text|upload)08 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436468#12325202 (10BCornwall) Yay, thanks for the help deploying it. Regarding tofu integration, I'm not see... [20:56:08] By the way, https://phabricator.wikimedia.org/T436472 is also awaiting your input - probably a tiny bit of remaining work. deployment-prep's puppetserver is complaning about stale certs for the Acme-chief hosts. Should be sufficient to https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/Decommission_an_instance issue two commands on the puppetserver for each decommissioned Acme-chief host. [20:59:18] ah, thanks for that [20:59:32] Dragging the new hosts into Tofu is probably https://phabricator.wikimedia.org/T394316. More than fine to postpone that, needs some automation to import existing resources into Tofu state. Getting rid of the Bullseye VMs is your accomplishment here :-) [21:01:49] Southparkfan: Thanks for the poke, I've deactivated/cleaned the acme-chief hosts. I'll wait on cleaning up the cache-* prefixes for a day or two juuuuust in case [21:02:08] Alright [21:02:20] unless you're confident enough to do it now :P [21:02:21] Great work, cleaning 'em up [21:02:37] thanks for all the help :) [21:02:53] nice job indeed brett! thanks Southparkfan for all your help and work as well <3 [21:02:57] Meh. Being careful is fine [21:03:03] would love to use tf, just a matter of priorities at present :) [21:03:27] old cache hosts are shut down for now [21:04:14] hmmmm errors [21:04:30] 503s, ugh [21:05:14] Almost breaking all of authoritative DNS was a reason for me to exercise more due diligence when performing changes, even in the Beta Cluster [21:05:25] O.O [21:05:31] (was really not my intention...) [21:05:40] sukhe: cheers :) [21:06:23] Getting no backend errors on beta now [21:07:05] Southparkfan: Was there a step I missed? [21:07:44] Not as far as I know. `deployment-mediawiki16`'s Apache is reachable from `deployment-cp-text09`... [21:09:01] ah, https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/cd96a63126036efeaa8169a8ecb6294da5e1ee01%5E%21/ [21:09:10] I killed one two many IPs, I think [21:09:39] oh, no, that should be right... [21:11:52] Yep. I have no clue... [21:13:23] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12325231 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2003 for host cp6001.drmrs.wmnet with OS trixie [21:14:02] oh, ats isn't running [21:15:44] Haha, sounds plausible [21:21:06] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12325240 (10BCornwall) Also note that I had to upgrade the BIOS on cp6001 to get it to even properly set the bios values, so if the reprovisioning fails, perhaps an upgrade is in order... [21:25:17] ats running isn't fixing it, so I'm really confused on what's going on [21:25:34] it was working before, so surely it was something I did on the cleanup [21:28:01] Can't find really useful ATS error logging, do we have that? [21:29:00] we turn logging off, so if it's not in the journal we need to stream the socket directly [21:29:30] Usually that's with 'fifo-log-tailer -socket /var/run/trafficserver/notpurge.pipe' but I see that fifo-log-tailer isn't available... [21:30:35] Yeah, because we set `profile::trafficserver::backend::monitor_enable: false` [21:31:13] must be a varnish config error if it's complaining about no backend? [21:31:33] Which skips `profile::trafficserver::logs`, which in turn skips `fifo_log_demux::instance` :/ [21:34:00] I manually installed it, fwiw [21:34:26] Aug 27 00:16:04 deployment-cp-text09 varnish-frontend[26279]: VCL compilation failed [21:34:30] I think you're at something [21:34:46] vcl-903b91bd-c3ff-46fc-ad8f-bccbd0d23326.be_deployment_cache_text08_deployment_prep_eqiad1_wikimedia_cloud probe 0/5 sick Tue, 15 Sep 2026 21:18:41 GMT [21:34:50] Oh, nevermind, sorry. That's old :) [21:35:15] Ah, right [21:35:28] And that list of backends is populated using `cache::nodes`, right? [21:36:24] yeah [21:37:09] aha [21:37:12] Right! The puppet prefix `deployment-cp-text` in Horizon still lists deployment-cache-text08.deployment-prep.eqiad1.wikimedia.cloud [21:37:14] it's in the stupid prefix puppet [21:37:18] jinx :) [21:37:26] I should just remove that, yeah? [21:37:39] the project puppet has it already [21:37:46] Go for it [21:38:21] killing cache::text::nodes which seems to be a legacy hiera [21:38:25] as well [21:38:41] Yes, that's been deprecated in 2019(!) - e6505be2020af32c79ab2cebb30b1d8ee22c3136 [21:38:51] err, since [21:39:00] Yet another well overdue change... [21:39:31] puppet's running, I'm seeing good diffs... [21:39:38] and we're back in business [21:39:39] phew [21:40:09] sorry for the extra confusion [21:40:09] Nice work :) [21:40:12] Thanks for the help! [21:40:31] I owe you something, I guess [21:40:51] not at all, I owe you something, would've been much more painful without you :) [21:42:18] Haha. Second pair of eyes is often useful. [21:43:19] Well, now it's definitely fine to leave the old cache proxies lingering for a little longer... [21:44:06] indeed [21:44:22] glad I shut em down, though, it was good to see that the backend was still pointing to them [21:45:25] I am also curious to see if the IPv6 move will break anything [21:48:09] Yeah [21:48:32] To date, https://gerrit.wikimedia.org/r/c/operations/puppet/+/1333288 is the only change I had to make in Beta Cluster to add dual-stack support [21:49:13] here's hoping! [21:50:37] Fortunately, we are driving IPv6 adoption.. [21:58:03] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12325349 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2003 for host cp6001.drmrs.wmnet with OS trixie completed: - cp6001 (**WARN**) - Downtimed on Icinga/A... [22:06:48] Alright, time to catch up on sleep. Thanks for the help on the new cache proxies! Cya [22:34:11] 06Traffic, 13Patch-For-Review: Reimage cp-upload nodes as cp-text nodes - https://phabricator.wikimedia.org/T436363#12325503 (10BCornwall) [22:34:12] \o [22:36:30] FIRING: [2x] LibericaDiffFPCheck: Liberica instance lvs6001:9100 control plane status doesn't match with forwarding plane status - https://wikitech.wikimedia.org/wiki/Liberica#LibericaDiffFPCheck - https://alerts.wikimedia.org/?q=alertname%3DLibericaDiffFPCheck [22:46:30] RESOLVED: LibericaDiffFPCheck: Liberica instance lvs6001:9100 control plane status doesn't match with forwarding plane status - https://wikitech.wikimedia.org/wiki/Liberica#LibericaDiffFPCheck - https://grafana.wikimedia.org/d/fa4de97a-7114-48c7-a91a-f56089ef554f/liberica?var-site=drmrs&var-instance=lvs6001 - https://alerts.wikimedia.org/?q=alertname%3DLibericaDiffFPCheck [23:06:26] 06Traffic: migrate traffic host pools to a standard weight numbering system - https://phabricator.wikimedia.org/T438098 (10BCornwall) 03NEW