[01:25:17] 10Traffic, 10SRE, 10Patch-For-Review: Test envoyproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T271421 (10AntiCompositeNumber) Another report of user-facing impact in `#mediawiki` from someone using the w3m browser: https://wm-bot.wmcloud.org/logs/%23mediawiki/20220... [03:21:18] 10Traffic, 10SRE, 10envoy, 10serviceops: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10RLazarus) Hm, reprepro has only has 1.15.4 in wikimedia-stretch, compared to 1.15.5 in buster and bullseye. I assume that's an oversight and not an intentional holdback, but so far I h... [03:38:14] 10Traffic, 10SRE, 10WMF-Legal, 10Performance-Team (Radar), 10Privacy: Consider disabling Chrome Lite pages for Wikipedia on Chrome on mobile with Cache-Control: no-transform - https://phabricator.wikimedia.org/T218618 (10dr0ptp4kt) Before we touch `no-transform` I'm requesting that @marayana and team mak... [05:23:56] (EdgeTrafficDrop) firing: 56% request drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqiad&var-cache_type=text - https://alerts.wikimedia.org [05:28:56] (EdgeTrafficDrop) resolved: 62% request drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqiad&var-cache_type=text - https://alerts.wikimedia.org [06:44:56] (EdgeTrafficDrop) firing: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [06:49:56] (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [06:52:56] (EdgeTrafficDrop) firing: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [07:02:56] (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [07:49:22] 10Traffic, 10SRE, 10ops-eqiad: asw2-b-eqiad:xe-2/0/3 interface errors (lvs1015) - https://phabricator.wikimedia.org/T300703 (10Peachey88) [09:16:32] 10Traffic, 10SRE, 10Patch-For-Review: Test envoyproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T271421 (10Vgutierrez) [09:17:04] 10Traffic, 10SRE: Problem loading thumbnail images due to Envoy (HTTP/1.0 clients getting '426 Upgrade Required') - https://phabricator.wikimedia.org/T300366 (10Vgutierrez) 05Open→03In progress p:05Triage→03Medium [09:18:59] 10Traffic, 10SRE, 10ops-eqiad: asw2-b-eqiad:xe-2/0/3 interface errors (lvs1015) - https://phabricator.wikimedia.org/T300703 (10ayounsi) [09:33:58] 10Traffic, 10SRE, 10ops-eqiad: asw2-b-eqiad:xe-2/0/3 interface errors (lvs1015) - https://phabricator.wikimedia.org/T300703 (10fgiunchedi) I've chatted with @ayounsi and now we have T300836 to track more effective tasks for inbound interface errors [09:43:33] 10Traffic, 10SRE, 10envoy, 10serviceops: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm) >>! In T300324#7673609, @RLazarus wrote: > Hm, reprepro has only has 1.15.4 in wikimedia-stretch, compared to 1.15.5 in buster and bullseye. I assume that's an oversight and... [09:48:30] 10Traffic, 10SRE, 10Patch-For-Review: Test envoyproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T271421 (10Vgutierrez) [09:49:32] 10Traffic, 10SRE, 10Patch-For-Review: Problem loading thumbnail images due to Envoy (HTTP/1.0 clients getting '426 Upgrade Required') - https://phabricator.wikimedia.org/T300366 (10Vgutierrez) 05In progress→03Resolved a:03Vgutierrez Thanks for reporting this issue, this unveiled a bug in our envoyproxy... [15:23:25] 10netops, 10Infrastructure-Foundations: Remove static routes for LVS VIPs from core routers - https://phabricator.wikimedia.org/T300877 (10cmooney) p:05Triage→03Low [15:48:08] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Remove static routes for LVS VIPs from core routers - https://phabricator.wikimedia.org/T300877 (10ayounsi) [21:25:52] 10Traffic, 10SRE, 10envoy, 10serviceops: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10RLazarus) >>! In T300324#7673609, @RLazarus wrote: > reprepro has only has 1.15.4 in wikimedia-stretch, compared to 1.15.5 in buster and bullseye. Correction: 1.15.5 is only in buster... [23:01:01] 10Traffic, 10SRE: Problem loading thumbnail images due to Envoy (HTTP/1.0 clients getting '426 Upgrade Required') - https://phabricator.wikimedia.org/T300366 (10Esanders) I still can't see thumbnails locally and on Patch demo. [23:05:22] 10Traffic, 10SRE: Problem loading thumbnail images due to Envoy (HTTP/1.0 clients getting '426 Upgrade Required') - https://phabricator.wikimedia.org/T300366 (10Esanders) 05Resolved→03Open [23:05:27] 10Traffic, 10SRE, 10Patch-For-Review: Test envoyproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T271421 (10Esanders)