[10:05:34] 06Traffic: Kapow score support on cache hosts - https://phabricator.wikimedia.org/T435799 (10Fabfur) 03NEW [10:46:17] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 06SRE: AlertLintProblem for TransitBGPDown check - https://phabricator.wikimedia.org/T435801 (10cmooney) 03NEW p:05Triage→03Low [11:03:21] 06Traffic, 10Internet-Archive, 06SRE, 07Regression: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743#12246220 (10Aklapper) Has this been reported to archive.org? What does `server does not respond.` mean exactly? I have a hard time to imagine that Commons or English Wikipedia... [12:07:26] https://usercontent.irccloud-cdn.com/file/Jqc7NdH7/grafik.png [12:07:32] it'll go up though [13:25:55] 06Traffic, 10Internet-Archive, 06SRE, 07Regression: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743#12246732 (10AlexisJazz) >>! In T435743#12246220, @Aklapper wrote: > Has this been reported to archive.org? What does `server does not respond.` mean exactly? I have a hard time... [13:41:50] 06Traffic, 10Internet-Archive, 06SRE, 07Regression: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743#12246796 (10TheDJ) Maybe @cyberpower678 can help find the right contacts [13:43:16] 06Traffic, 10Internet-Archive, 06SRE, 07Regression: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743#12246803 (10ssingh) We are looking at the internal traffic logs and checking if archive.org is matching some other rule, or one of their own rules (and if something changed the... [14:11:56] 06Traffic, 10Internet-Archive, 06SRE, 07Regression: archive.org cannot reach Wikimedia - https://phabricator.wikimedia.org/T435743#12246915 (10Fabfur) from a rapid look at our logs, we're not currently blocking archive.org bot and trying to save a couple of pages from different wiki (enwiki, itwiki) works... [14:38:39] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 06SRE: AlertLintProblem for TransitBGPDown check - https://phabricator.wikimedia.org/T435801#12247067 (10tappof) The alerts come from Pint, a linter/validator for Prometheus. It just says that the expression defined in the TransitBGPDown rul... [14:39:31] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 06SRE: AlertLintProblem for TransitBGPDown check - https://phabricator.wikimedia.org/T435801#12247073 (10tappof) [14:57:32] 10netops, 06SRE Observability: Prometheus rule evaluation failures (instance titan1001) - https://phabricator.wikimedia.org/T435494#12247266 (10LSobanski) [14:58:40] 10netops, 06SRE Observability: Prometheus rule evaluation failures (instance titan1001) - https://phabricator.wikimedia.org/T435494#12247271 (10cmooney) Are there still any problems here folks? I'm hoping it cleared up when we shifted the traffic away from the problematic link on Thursday? [16:25:26] leaving this here to come back to it when picking the next wiki: https://usercontent.irccloud-cdn.com/file/sX31D4vE/grafik.png [16:39:14] ok [17:06:27] 06Traffic, 06Commons: Varnish cache purges of file assets sometimes seem to be failing, at least sometimes - https://phabricator.wikimedia.org/T435283#12248179 (10BBlack) >>! In T435283#12240228, @Bawolff wrote: > Note how for thumbnails, it is purging urls without the utm query parameters (e.g. https://upload... [17:28:14] hello traffic friends - any concerns or conflicts if I deploy an ATS Lua change [0] in a little bit? this is an improved version of the patch I was originally going to deploy last Thursday :) [17:28:14] [0] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1327621 [17:33:53] swfrench-wmf: Looking through it seems the first two hunks are what's changed. It seems reasonable though I've not spent much time in the lua ats systems. In any case, I'm familiarized with what you're trying to do, so go for it [17:34:41] brett: ah, thanks! sorry, I didn't mean to add review to your plate - I was mainly asking in order not to clash with other work you might have happening :) [17:34:51] but thank you for taking a look as well! [17:35:57] Really, I *should* be looking at all of these things, seeing as it's my team's ownership. So consider it a weak attempt at doing my job properly :P [17:37:01] lovely commit message, by the by :) [17:40:18] haha, fair enough, and thank you [17:45:37] 10netops, 06Infrastructure-Foundations, 10Observability-Metrics, 06SRE: Expand blackbox icmp probes to ping specific router interfaces/circuits - https://phabricator.wikimedia.org/T435855 (10cmooney) 03NEW p:05Triage→03High [17:46:45] 10netops, 06Infrastructure-Foundations, 10Observability-Metrics, 06SRE: Expand blackbox icmp probes to ping specific router interfaces/circuits - https://phabricator.wikimedia.org/T435855#12248368 (10cmooney) [19:05:00] 06Traffic, 06Commons: Varnish cache purges of file assets sometimes seem to be failing, at least sometimes - https://phabricator.wikimedia.org/T435283#12248637 (10ssingh) [Related: T425216] [19:22:32] 06Traffic, 06MediaWiki-Media-Platform-Team, 10Thumbor, 13Patch-For-Review: Add `thumb_generated` from X-Analytics to webrequest_sampled_live - https://phabricator.wikimedia.org/T435634#12248682 (10Ladsgroup) a:03Ladsgroup [20:05:17] 06Traffic, 10Beta-Cluster-Infrastructure, 06SRE: Rename deployment-cache-(text|upload)0x to deployment-cp-(text|upload)0x - https://phabricator.wikimedia.org/T280393#12248905 (10bd808) [20:06:04] 06Traffic, 10Beta-Cluster-Infrastructure, 06SRE: Rename deployment-cache-(text|upload)0x to deployment-cp-(text|upload)0x - https://phabricator.wikimedia.org/T280393#12248929 (10bd808) [20:09:50] 06Traffic, 10Maps (Kartographer): Geoshape rate limiting breaks the live maps - https://phabricator.wikimedia.org/T435741#12248941 (10CDanis) I can reproduce. The edge ratelimit in question has been in place since 2019, not something new. I think the proper fix would be within the [[ https://gerrit.wikimedia... [21:40:26] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 06SRE: AlertLintProblem for TransitBGPDown check - https://phabricator.wikimedia.org/T435801#12249177 (10cmooney) Ok thanks that's a major issue with our stats pipeline then. The cr2-eqord we can ignore, if we have no stats for BGP that's a... [21:51:47] 06Traffic, 06Data-Platform-SRE, 10Wikidata, 10WMDE Analytics, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#12249259 (10Ahoelzl) 05Open→03Resolved [21:52:20] 06Traffic, 06Data-Persistence, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 13Patch-For-Review: Data Lake - mediarequests - adapt HQL query to include counting mediarequests from webrequest_text - https://phabricator.wikimedia.org/T431475#12249271 (10Ahoelzl) 05Open→03Resolved [21:55:45] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 06SRE: AlertLintProblem for TransitBGPDown check - https://phabricator.wikimedia.org/T435801#12249291 (10cmooney) 05Open→03Resolved a:03cmooney Actually I worked it out. We stopped getting stats on July 15 when we upgraded the rou... [23:00:59] 06Traffic, 06DC-Ops, 10ops-drmrs: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12249532 (10RobH) DHL called me back today, they're still trying to get in contact with the local field office for the drop off to find out what is going on with the package...