[01:03:21] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade management routers and switches to Junos 21 - https://phabricator.wikimedia.org/T316529 (10Papaul) a:03Papaul [01:08:03] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade management routers and switches to Junos 21 - https://phabricator.wikimedia.org/T316529 (10Papaul) [08:29:19] 10Traffic, 10SRE, 10Upstream: ATS wrongly parses requests without a leading / - https://phabricator.wikimedia.org/T317660 (10Vgutierrez) [08:30:04] 10Traffic, 10SRE, 10Upstream: ATS wrongly parses requests without a leading / - https://phabricator.wikimedia.org/T317660 (10Vgutierrez) Making the task public after cleaning IP addresses from the original request that helped detecting the issue and after checking with upstream that this isn't a security bug [09:49:45] 10Traffic: CDN doesn't validate request-target - https://phabricator.wikimedia.org/T318676 (10Vgutierrez) [09:50:12] 10Traffic: CDN doesn't validate request-target - https://phabricator.wikimedia.org/T318676 (10Vgutierrez) [09:51:18] 10Traffic, 10SRE: CDN doesn't validate request-target - https://phabricator.wikimedia.org/T318676 (10Vgutierrez) p:05Triage→03Medium [09:57:59] 10Traffic, 10SRE: CDN doesn't validate request-target - https://phabricator.wikimedia.org/T318676 (10Vgutierrez) Apparently varnish supports the absolute-URI form for non CONNECT requests. This has been introduced a long time ago in https://gerrit.wikimedia.org/r/c/operations/puppet/+/275474. @BBlack do you ha... [12:13:56] (HAProxyEdgeTrafficDrop) firing: 61% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [12:18:56] (HAProxyEdgeTrafficDrop) resolved: 64% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [14:16:41] 10Traffic, 10SRE, 10Patch-For-Review: Create program to interact with Atlas RIPE API - https://phabricator.wikimedia.org/T315536 (10BCornwall) While we have https://gerrit.wikimedia.org/r/c/operations/software/latency-measurement/+/833848 available for review, I hear that there'd be pushback for not having a... [14:30:09] 10Traffic, 10InternetArchiveBot, 10SRE: IABot is encountering 429 on Wikimedia Production - https://phabricator.wikimedia.org/T318065 (10Joe) Do you happen to have any further detail on the response headers and body you get whenever you receive a 429 response? it would help us identify which layer is returni... [14:35:49] 10Traffic, 10InternetArchiveBot, 10SRE: IABot is encountering 429 on Wikimedia Production - https://phabricator.wikimedia.org/T318065 (10Cyberpower678) >>! In T318065#8265005, @Joe wrote: > Do you happen to have any further detail on the response headers and body you get whenever you receive a 429 response?... [14:40:33] 10Traffic, 10InternetArchiveBot, 10SRE: IABot is encountering 429 on Wikimedia Production - https://phabricator.wikimedia.org/T318065 (10Cyberpower678) Actually, I have some left from intentionally hitting them while testing the bot yesterday. ` array(37) { ["url"]=> string(131) "https://en.wikipedia.o... [14:45:02] 10Traffic, 10InternetArchiveBot, 10SRE: IABot is encountering 429 on Wikimedia Production - https://phabricator.wikimedia.org/T318065 (10Cyberpower678) The bot has two IPs it works from. # 185.15.56.22 # 185.15.56.29 [15:11:09] 10Traffic, 10InternetArchiveBot, 10SRE: IABot is encountering 429 on Wikimedia Production - https://phabricator.wikimedia.org/T318065 (10Vgutierrez) > This ticket is two-fold. The first is a request for SRE to provide logs regarding queries originating from IABot, easily identified from the UA. @Cyberpower67... [15:24:43] 10Traffic, 10InternetArchiveBot, 10SRE: IABot is encountering 429 on Wikimedia Production - https://phabricator.wikimedia.org/T318065 (10Vgutierrez) As a reference, this change in behavior has been triggered by https://gerrit.wikimedia.org/r/c/operations/puppet/+/677872 [15:25:41] 10Traffic, 10InternetArchiveBot, 10SRE: IABot is encountering 429 on Wikimedia Production - https://phabricator.wikimedia.org/T318065 (10Cyberpower678) >>! In T318065#8265171, @Vgutierrez wrote: >> This ticket is two-fold. The first is a request for SRE to provide logs regarding queries originating from IABo... [15:29:53] 10Traffic, 10InternetArchiveBot, 10SRE: IABot is encountering 429 on Wikimedia Production - https://phabricator.wikimedia.org/T318065 (10Vgutierrez) > Without specific logs, I can't really assess if these aggressive requests can be optimized. I would recommend generating those logs on the IABot side [15:30:56] 10Traffic, 10InternetArchiveBot, 10SRE: IABot is encountering 429 on Wikimedia Production - https://phabricator.wikimedia.org/T318065 (10Cyberpower678) >>! In T318065#8265227, @Vgutierrez wrote: >> Without specific logs, I can't really assess if these aggressive requests can be optimized. > I would recommend... [15:33:15] 10Traffic, 10InternetArchiveBot, 10SRE: IABot is encountering 429 on Wikimedia Production - https://phabricator.wikimedia.org/T318065 (10Cyberpower678) >>! In T318065#8265199, @Vgutierrez wrote: > As a reference, this change in behavior has been triggered by https://gerrit.wikimedia.org/r/c/operations/puppet... [15:34:15] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: Q1:rack/setup/install cp - https://phabricator.wikimedia.org/T317244 (10BBlack) Copying over from T317249#8262220 - This is the replacement mapping of nodes + disks: | cp nodes | Current | Replacement | Disks | text | 21-26, 33, 34 | 37-44 | Single NVME | upload |... [15:54:30] 10Traffic, 10InternetArchiveBot, 10SRE: IABot is encountering 429 on Wikimedia Production - https://phabricator.wikimedia.org/T318065 (10Cyberpower678) @Vgutierrez is there an explanation somewhere why the Cloud VPS IP range was removed from this list? Is it possible to add IABot IPs back on until we can ge... [15:58:39] 10Traffic, 10InternetArchiveBot, 10SRE: IABot is encountering 429 on Wikimedia Production - https://phabricator.wikimedia.org/T318065 (10Vgutierrez) >>! In T318065#8265344, @Cyberpower678 wrote: > @Vgutierrez is there an explanation somewhere why the Cloud VPS IP range was removed from this list? Is it poss... [16:03:04] 10Traffic, 10InternetArchiveBot, 10SRE: IABot is encountering 429 on Wikimedia Production - https://phabricator.wikimedia.org/T318065 (10BBlack) >>! In T318065#8265200, @Cyberpower678 wrote: > IABot workers run independently of each other. Each worker runs on a single wiki and minds it's own business. So t... [16:06:41] 10Traffic, 10InternetArchiveBot, 10SRE: IABot is encountering 429 on Wikimedia Production - https://phabricator.wikimedia.org/T318065 (10Cyberpower678) >>! In T318065#8265366, @BBlack wrote: >>>! In T318065#8265200, @Cyberpower678 wrote: >> IABot workers run independently of each other. Each worker runs on... [16:13:45] 10Traffic, 10InternetArchiveBot, 10SRE: IABot is encountering 429 on Wikimedia Production - https://phabricator.wikimedia.org/T318065 (10ayounsi) >>! In T318065#8265347, @Vgutierrez wrote: > that would be a question for @ayounsi / @cmooney from the netops team and/or @Andrew from WMCS Context is in T265864,... [16:17:52] 10Traffic, 10InternetArchiveBot, 10SRE: IABot is encountering 429 on Wikimedia Production - https://phabricator.wikimedia.org/T318065 (10Cyberpower678) >>! In T318065#8265422, @ayounsi wrote: >>>! In T318065#8265347, @Vgutierrez wrote: >> that would be a question for @ayounsi / @cmooney from the netops team... [16:18:13] 10Traffic, 10InternetArchiveBot, 10SRE: IABot is encountering 429 on Wikimedia Production - https://phabricator.wikimedia.org/T318065 (10BBlack) >>! In T318065#8265397, @Cyberpower678 wrote: >>>! In T318065#8265366, @BBlack wrote: >>>>! In T318065#8265200, @Cyberpower678 wrote: >>> IABot workers run independ... [16:23:51] 10Traffic, 10InternetArchiveBot, 10SRE: IABot is encountering 429 on Wikimedia Production - https://phabricator.wikimedia.org/T318065 (10Cyberpower678) >>! In T318065#8265446, @BBlack wrote: >>>! In T318065#8265397, @Cyberpower678 wrote: >>>>! In T318065#8265366, @BBlack wrote: >>>>>! In T318065#8265200, @Cy...