[00:27:27] 06Traffic, 06SRE, 07Documentation: TLS 1.2 on Wikimedia DNS DoH resolver not working - https://phabricator.wikimedia.org/T415449#11551498 (10Naruse_shiroha) Okay, updated it in https://meta.wikimedia.org/w/index.php?title=Wikimedia_DNS&diff=prev&oldid=29976804. [00:29:44] 06Traffic, 06SRE, 07Documentation: Documentation error about TLS 1.2 on Wikimedia DNS DoH on metawiki - https://phabricator.wikimedia.org/T415449#11551499 (10Naruse_shiroha) 05Open→03Resolved [01:01:01] 06Traffic, 06SRE: TCP Fast Open not working since at least December 2025 - https://phabricator.wikimedia.org/T415454 (10Cuthead) 03NEW [01:04:57] 06Traffic, 06SRE: TCP FastOpen not working since at least December 2025 - https://phabricator.wikimedia.org/T415454#11551516 (10Cuthead) [07:43:38] 06Traffic, 06SRE: TCP FastOpen not working since at least December 2025 - https://phabricator.wikimedia.org/T415454#11551605 (10Bewfip) There are some mentions of TCP Fast Open in operations/puppet: https://codesearch.wmcloud.org/puppet/?q=tcp_fastopen . Though I don't know how the network stack here works. [07:55:34] 06Traffic, 06SRE: TCP FastOpen not working since at least December 2025 - https://phabricator.wikimedia.org/T415454#11551607 (10Cuthead) Just confirmed TFO on 3 AuthDNS is OK. ` 208.80.153.231 age 179.516sec fo_mss 1024 fo_cookie 2961b389e2ec5b93 source 192.168.1.141 208.80.154.238 age 236.896sec fo_mss 1024... [08:43:35] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11551645 (10Ladsgroup) The URL to our thumbnails is not an stable API‌ and shouldn't be treated as such. The actual APIs return URL to... [09:13:05] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11551665 (10Tacsipacsi) I see. However, people //have// been treating it as a stable API, and there are also just too many places wher... [22:30:11] 06Traffic, 10DNS, 06SRE, 06Abstract Wikipedia team (26Q3 (Jan–Mar)), and 2 others: Set up DNS for abstract.wikipedia.org to be recognised - https://phabricator.wikimedia.org/T411724#11552111 (10MLechvien-WMF) [23:40:00] FIRING: PurgedHighEventLag: High event process lag with purged on cp3069:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=esams%20prometheus/ops&var-instance=cp3069 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [23:45:00] FIRING: [16x] PurgedHighEventLag: High event process lag with purged on cp3066:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [23:46:03] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-esams, 07Wikimedia-production-error: ESAMS 502 broken pipe connection issues - https://phabricator.wikimedia.org/T415473 (10AlexisJazz) 03NEW [23:59:06] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: ESAMS 502 broken pipe connection issues - https://phabricator.wikimedia.org/T415473#11552154 (10AlexisJazz)