[07:05:22] 06Traffic, 06Commons: Varnish cache purges of file assets sometimes seem to be failing, at least sometimes - https://phabricator.wikimedia.org/T435283 (10Bawolff) 03NEW [07:26:17] 06Traffic, 10hCaptcha, 06Product Safety and Integrity, 10ServiceOps-Mediawiki, and 2 others: memcached errors seen in hCaptcha health checks - https://phabricator.wikimedia.org/T430340#12229743 (10MLechvien-WMF) [07:26:30] 06Traffic, 10hCaptcha, 06Product Safety and Integrity, 10ServiceOps-Mediawiki, and 2 others: memcached errors seen in hCaptcha health checks - https://phabricator.wikimedia.org/T430340#12229745 (10MLechvien-WMF) p:05Triage→03Medium [07:54:37] 06Traffic, 06Machine-Learning-Team (Q1 FY2026-27): CDN caching request for TTS v1 audio files - https://phabricator.wikimedia.org/T434046#12229791 (10isarantopoulos) Hi Brett! Absolutely, let's meet. In the meantime the description in the parent task {T430676} might be able to provide answers to some of your q... [08:51:43] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 06SRE: Upgrade JunOS on pfw1-eqiad and pfw1-codfw - https://phabricator.wikimedia.org/T434865#12229923 (10cmooney) >>! In T434865#12228113, @cmooney wrote: > As discussed on irc I'll kick off the upgrade on pfw1-codfw at 09:00am UTC tomrorow (... [09:04:59] 06Traffic, 06Data-Persistence, 06MediaWiki-Media-Platform-Team, 13Patch-For-Review: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#12229948 (10SLyngshede-WMF) Rollout of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1298820 failed ` Aug 19 08:26:32 cp... [11:28:27] 06Traffic, 06Data-Persistence, 06MediaWiki-Media-Platform-Team, 13Patch-For-Review: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#12230651 (10SLyngshede-WMF) The traffic frontend now seems to work, but ATS fails to fetch the thumbnails from Swift. ` 202608... [12:23:10] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 06SRE: Upgrade JunOS on pfw1-eqiad and pfw1-codfw - https://phabricator.wikimedia.org/T434865#12230878 (10cmooney) As per conversation with @Jgreen on irc we will do the eqiad pair at 09:00UTC tomorrow morning (Thurs 20th). [13:08:10] cjd91: I see a sre.cdn.roll-upgrade-ats cookbook that just started for cp-codfw. just to confirm that will either be complete by 13:30 (22 minutes from now) or it is acceptable for that to fail then? [13:09:39] yeah, we should probably stop the cookbooks cjd91 ^ (my bad, I also missed it) [13:10:17] swfrench-wmf: ok, stopping them now. sorry about that [13:10:32] swfrench-wmf: I have one cookbook running that should be done by then [13:10:40] * swfrench-wmf thumbs up [13:10:42] thank you both! [13:10:53] just don't wanna break your stuff :) [14:47:34] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12231680 (10cmooney) >>! In T434806#12224818, @ayounsi wrote: > We should reach out to the BIRD team, even if there is a race condition, BIRD's BFD pr... [14:48:34] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12231685 (10cmooney) Small update. We merged the patch to only start the anycast-healthchecker service (and thus BIRD) after the network-online targe... [15:12:00] FIRING: PurgedHighEventLag: High event process lag with purged on cp7008:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=magru%20prometheus/ops&var-instance=cp7008 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [15:17:00] RESOLVED: PurgedHighEventLag: High event process lag with purged on cp7008:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=magru%20prometheus/ops&var-instance=cp7008 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [16:36:24] swfrench-wmf: is it ok for me to restart the cookbooks? [16:37:04] cjd91: ah, yes! please feel free to start again. [16:40:35] thank you! [16:43:39] 06Traffic, 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-cache-text08 on project deployment-prep - https://phabricator.wikimedia.org/T435193#12232802 (10bd808) p:05Triage→03High The VCL compiler is not happy yet: `counterexample Error: '/usr/local/sbin/reload-vcl -n fronte... [16:49:03] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet broken on deployment-cache-text08 by disk space & Puppet VCL changes - https://phabricator.wikimedia.org/T435193#12232879 (10taavi) >>! In T435193#12232804, @bd808 wrote: > @BCornwall is this a magically secret prod-only file that we should be feature flagging t... [16:49:07] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet broken on deployment-cache-text08 by disk space & Puppet VCL changes - https://phabricator.wikimedia.org/T435193#12232881 (10bd808) [16:57:07] 06Traffic, 06Data-Engineering: Turnilo: X-Is-Browser filter doesn't work (always HTTP 500) - https://phabricator.wikimedia.org/T435152#12232984 (10Ottomata) a:03JAllemandou [16:57:17] 06Traffic, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Turnilo: X-Is-Browser filter doesn't work (always HTTP 500) - https://phabricator.wikimedia.org/T435152#12232986 (10Ottomata) [17:43:39] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet broken on deployment-cache-text08 by disk space & Puppet VCL changes - https://phabricator.wikimedia.org/T435193#12233303 (10bd808) 05In progress→03Resolved Thanks for spotting that for me @taavi. Puppet run is clean now: `lang=shell-session root@deploy... [18:31:00] FIRING: [3x] PurgedHighEventLag: High event process lag with purged on cp7008:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [18:31:46] brett: cjd91: can we look into why cp7008 had this fired twice now? [18:31:53] it resolved previously [18:32:19] hmm I suspect it is a wider kafka problem [18:32:37] see email to sre-traffic@ cp7013 is also affected [18:36:00] RESOLVED: [3x] PurgedHighEventLag: High event process lag with purged on cp7008:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [18:39:00] FIRING: PurgedHighEventLag: High event process lag with purged on cp7008:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=magru%20prometheus/ops&var-instance=cp7008 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [18:44:00] RESOLVED: PurgedHighEventLag: High event process lag with purged on cp7008:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=magru%20prometheus/ops&var-instance=cp7008 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [18:45:26] 06Traffic, 06Data-Persistence, 06MediaWiki-Media-Platform-Team, 13Patch-For-Review: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#12233592 (10Ladsgroup) Overall works fine, there is one issue though: - https://thumb.wikimedia.org/wikipedia/test/thumb/9/92/7... [19:04:34] FYI, testwiki is now using thumb.wikimedia.org local files don't work but otherwise look good [19:04:48] patch to fix it https://gerrit.wikimedia.org/r/c/operations/puppet/+/1327175 [19:04:55] yay! [19:06:28] Amir1: I'll roll it out now [19:06:37] Thank you <3 [19:07:05] After an hour or so, I switch mediawiki.org as well, unless people want to wait [19:07:45] up to you! [19:16:06] Amir1: rolled out the regex lossening [19:16:49] thank you! [21:10:58] 10netops, 06Traffic, 06Infrastructure-Foundations, 10ops-eqsin: eqsin switch migration - 2026-08-26 @ 08:00 UTC - https://phabricator.wikimedia.org/T435406 (10RobH) 03NEW [21:12:32] 10netops, 06Traffic, 06Infrastructure-Foundations, 10ops-eqsin: eqsin switch migration - 2026-08-26 @ 08:00 UTC - https://phabricator.wikimedia.org/T435406#12234210 (10RobH) p:05Triage→03High a:03ssingh @ssingh, I wanted to get your sign off on this for #traffic before I fully commit with Jin. We'd... [23:16:45] mediawiki.org is now also migrated. Tomorrow I will switch fawiki and next week dewiki [23:25:27] 10Domains, 07HTTPS, 10DNS, 06SRE, 06Traffic-Icebox: Merge Wikipedia subdomains into one, to discourage censorship - https://phabricator.wikimedia.org/T215071#12234740 (10BCornwall) I also think this can be closed as there aren't any plans (that I know of) to remove the lang subdomains. mobile domains hav... [23:29:53] hooray!