[06:11:09] FIRING: [2x] LVSHighCPU: The host lvs1017:9100 has at least its CPU 22 saturated - https://bit.ly/wmf-lvscpu - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [06:16:09] RESOLVED: [2x] LVSHighCPU: The host lvs1017:9100 has at least its CPU 22 saturated - https://bit.ly/wmf-lvscpu - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [07:52:40] 10netops, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-08-07 - 2026-08-28), 07Kubernetes: Calico IPv4 block exhaustion on dse-k8s cluster, blocking new node provisioning - https://phabricator.wikimedia.org/T429773#12219894 (10Gehel) Moving to Blocked, work is happening on the subtask. [10:58:33] Hii, cross-posting what I said in engineering-all since I think this impacts you the most. We just enabled lazy loading of images on desktop (https://phabricator.wikimedia.org/T148047) which means traffic to images will shrink a bit (after caches expire). Example https://w.wiki/TZYY [10:59:43] ack, tnix [12:15:08] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, and 2 others: Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#12221251 (10JMeybohm) [12:50:24] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade JunOS on pfw1-eqiad and pfw1-codfw - https://phabricator.wikimedia.org/T434865#12221432 (10Jgreen) @cmooney this happens to be a FR maintenance week, is it feasible to plan it sometime tomorrow-Friday? [12:50:56] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 06SRE: Upgrade JunOS on pfw1-eqiad and pfw1-codfw - https://phabricator.wikimedia.org/T434865#12221433 (10Jgreen) [13:41:08] Hey Traffic, I had a question about hieradata for our cirrus (opensearch) hosts. Per T387569 we've created host hieradata for each host (example: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/hosts/cirrussearch2101.yaml) . But the service is no longer `opensearch_1`, it's now `opensearch_2`. Is that gonna cause a problem with LVS? [13:41:09] T387569: Update Elastic puppet code to filter LVS config based on cluster membership - https://phabricator.wikimedia.org/T387569 [13:41:26] Basically I'm asking if we need this hieradata anymore post-IPIP migration [13:41:49] inflatador: not really no, it should be fine [13:41:56] what sort of problem do you anticipate I guess? [13:42:25] sukhe it's something I completely missed during the 1->2 migration in June, so I imagine the problems would've happened already ;) [13:43:07] right now I'm just cleaning up any references to opensearch 1.x and those files showed up in a ripgrep [13:43:33] inflatador: so the service is already opensearch_2? [13:43:57] Yes, it's been that way for 2 months or so [13:44:57] the primary purpose of that is how it translates to the safe_service_restart bits [13:45:13] so yeah, there could have been impact in that respect, but hard to trace it now [13:45:25] let's update that now I guess? [13:45:42] ACK, I can update instead of remove. Will get a patch up shortly [13:45:58] thanks [14:19:13] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12222058 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host dns1006.wikimedia.org with OS trixie [14:19:24] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12222063 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host dns2006.wikimedia.org with OS trixie [14:19:40] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12222068 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host dns4004.wikimedia.org with OS trixie [14:31:19] 10netops, 06Infrastructure-Foundations: codfw: upgrade Nokia E/F switches to SR-Linux 26.7 - https://phabricator.wikimedia.org/T434070#12222130 (10cmooney) p:05Triage→03Medium [14:32:03] 10netops, 06Infrastructure-Foundations: ulsfo: upgrade switches to SR-Linux 26.7 - https://phabricator.wikimedia.org/T434068#12222135 (10cmooney) p:05Triage→03Medium [14:53:21] 06Traffic: Handle exception raised when OCSP response is an error - https://phabricator.wikimedia.org/T435104 (10CDobbins) 03NEW [14:59:48] sukhe just following up from before, atsukoito was nice enough to review https://gerrit.wikimedia.org/r/c/operations/puppet/+/1326304 which I just deployed. So we should be good as far as the safe service scripts now [15:00:25] inflatador: thanks, yes that should be it I think (and hold me responsible if it was not :) [15:13:28] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12222493 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host dns4004.wikimedia.org with OS trixie executed with errors: - dns4004 (**FAIL**) - Downtimed on Icinga/Alertmanager -... [15:19:06] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12222522 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host dns4004.wikimedia.org with OS trixie [15:19:24] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12222524 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host dns1006.wikimedia.org with OS trixie completed: - dns1006 (**WARN**) - Downtimed on Icinga/Alertmanager - Disabled Pu... [15:26:01] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12222565 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host dns2006.wikimedia.org with OS trixie completed: - dns2006 (**WARN**) - Downtimed on Icinga/Alertmanager - Disabled Pu... [16:43:41] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12223079 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host dns4004.wikimedia.org with OS trixie completed: - dns4004 (**PASS**) - Removed from Puppet and PuppetDB if present and... [21:03:35] hello traffic friends - wanted to check in here in case things have changed: we have some stale cached content for [0] (updated earlier today) on a subset of cache-text hosts. is [1] still the correct the correct tool for a one-off purge like this? [21:03:35] [0] https://test.wikipedia.org/.well-known/apple-app-site-association [21:03:35] [1] https://wikitech.wikimedia.org/wiki/Kafka_HTTP_purging#One-off_purge [21:24:15] swfrench-wmf: AFAIK yes, that's correct [21:24:36] awesome, thanks brett :)