[07:06:44] 10Wikimedia-Apache-configuration, 10SRE, 10serviceops: Incorrect handling of ETags taking precedence over timestamps in conditional requests - https://phabricator.wikimedia.org/T320241 (10Ifrahkhanyaree) [09:11:50] 10Traffic, 10SRE: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10Fabfur) [09:49:28] fabfur: o/ you have now officially stepped into the varnishkafka's curse, welcome! [09:49:41] hey, just packaging [09:49:54] I'm not responsible for my actions! [09:49:56] :D [09:49:59] doesn't matter of course! [09:50:00] :D [09:51:31] really, still haven't chance to look at the code, just repackaging for bookworm [09:54:35] please don't :D [10:44:30] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [10:44:31] you're now the official maintainer [10:45:52] "uploader", please [10:46:03] :) [12:08:22] 10Traffic, 10Abstract Wikipedia team, 10WikiLambda: Varnish/ATS are occasionally responding to Wikifunctions object page reads with a 404 even though `cache;desc="pass"` is set on normal requests - https://phabricator.wikimedia.org/T343440 (10Jdforrester-WMF) [12:48:10] 10Traffic, 10Abstract Wikipedia team, 10SRE, 10WikiLambda: Varnish/ATS are occasionally responding to Wikifunctions object page reads with a 404 even though `cache;desc="pass"` is set on normal requests - https://phabricator.wikimedia.org/T343440 (10Vgutierrez) ` vgutierrez@carrot:~$ for i in {1..100}; do... [12:52:50] 10Traffic, 10Abstract Wikipedia team, 10MW-on-K8s, 10SRE, and 2 others: Varnish/ATS are occasionally responding to Wikifunctions object page reads with a 404 even though `cache;desc="pass"` is set on normal requests - https://phabricator.wikimedia.org/T343440 (10Clement_Goubert) a:03Clement_Goubert [12:54:43] 10Traffic, 10Abstract Wikipedia team, 10MW-on-K8s, 10SRE, and 2 others: Varnish/ATS are occasionally responding to Wikifunctions object page reads with a 404 even though `cache;desc="pass"` is set on normal requests - https://phabricator.wikimedia.org/T343440 (10Clement_Goubert) I'll prepare a patch to exc... [13:04:02] 10Traffic, 10Abstract Wikipedia team, 10MW-on-K8s, 10SRE, and 2 others: Varnish/ATS are occasionally responding to Wikifunctions object page reads with a 404 even though `cache;desc="pass"` is set on normal requests - https://phabricator.wikimedia.org/T343440 (10Jdforrester-WMF) p:05Triage→03High [13:13:26] 10Traffic, 10Abstract Wikipedia team, 10MW-on-K8s, 10SRE, and 2 others: mw-on-k8s responds 404 for Wikifunctions view pages - https://phabricator.wikimedia.org/T343440 (10Clement_Goubert) [13:19:14] 10Traffic, 10Abstract Wikipedia team, 10MW-on-K8s, 10SRE, and 3 others: mw-on-k8s responds 404 for Wikifunctions view pages - https://phabricator.wikimedia.org/T343440 (10Clement_Goubert) It's a missing rewrite rule in mediawiki::sites [14:54:39] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [15:20:58] 10Traffic, 10Abstract Wikipedia team, 10MW-on-K8s, 10SRE, and 2 others: mw-on-k8s responds 404 for Wikifunctions view pages - https://phabricator.wikimedia.org/T343440 (10Clement_Goubert) ` cgoubert@deploy2002:~$ curl -s --insecure -v -H "Host: www.wikifunctions.org" https://mwdebug.discovery.wmnet:4444/vi... [15:27:55] 10Traffic, 10Abstract Wikipedia team, 10MW-on-K8s, 10SRE, and 2 others: mw-on-k8s responds 404 for Wikifunctions view pages - https://phabricator.wikimedia.org/T343440 (10Clement_Goubert) 05Open→03Resolved ` ❯ for i in {1..100}; do curl -s -v https://www.wikifunctions.org/view/en/Z10000 -o /dev/null 2>... [15:30:47] 10Traffic, 10Abstract Wikipedia team, 10MW-on-K8s, 10SRE, and 2 others: mw-on-k8s responds 404 for Wikifunctions view pages - https://phabricator.wikimedia.org/T343440 (10Jdforrester-WMF) Thank you! [15:40:50] 10Traffic, 10SRE: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10Fabfur) [17:12:42] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp1[098-113] - https://phabricator.wikimedia.org/T342159 (10ssingh) >>! In T342159#9025176, @RobH wrote: > Please note parent task 341588 has the range of cp1[090-105] however, cp1090 is already live/in use. Additionally, we have 4 cp hosts fr... [18:27:29] 10Traffic: Investigate why Traffic SLO Grafana dashboard has negative values on combined SLI - https://phabricator.wikimedia.org/T341606 (10BCornwall) From discussion on IRC: ` I'm noticing that the SLO calculations on grafana seem a little off but I may be missing something. Does anyone have a definiti... [18:27:43] 10Traffic: Investigate why Traffic SLO Grafana dashboard has negative values on combined SLI - https://phabricator.wikimedia.org/T341606 (10BCornwall) [19:34:34] 10Traffic: Investigate why Traffic SLO Grafana dashboard has negative values on combined SLI - https://phabricator.wikimedia.org/T341606 (10herron) Thanks for creating a tracking task! Quickly adding my notes from looking into this: I created dashboard https://grafana.wikimedia.org/d/Uk3M_9C4k/haproxy-sli-scra... [19:37:18] 10Traffic: Investigate why Traffic SLO Grafana dashboard has negative values on combined SLI - https://phabricator.wikimedia.org/T341606 (10BCornwall) Based on the IRC conversation, wouldn't A have the same issues that we currently have? I'm not convinced that C is the best way forward since it would be sweepin... [19:38:44] 10Traffic: Investigate why Traffic SLO Grafana dashboard has negative values on combined SLI - https://phabricator.wikimedia.org/T341606 (10BCornwall) [20:38:02] 10Traffic: Investigate why Traffic SLO Grafana dashboard has negative values on combined SLI - https://phabricator.wikimedia.org/T341606 (10herron) >>! In T341606#9067700, @BCornwall wrote: > Based on the IRC conversation, wouldn't A have the same issues that we currently have? Today we use `1 - ( good / total... [20:46:47] 10Traffic: Investigate why Traffic SLO Grafana dashboard has negative values on combined SLI - https://phabricator.wikimedia.org/T341606 (10BCornwall) bad, good, and total are still not guaranteed to be synced up and are still suspect data, aren't they? It's also entirely possible that many bad requests could ex... [20:50:49] 10Traffic, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall) 05Open→03In progress