[08:01:56] (HAProxyEdgeTrafficDrop) firing: 67% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:06:56] (HAProxyEdgeTrafficDrop) resolved: 69% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:30:19] 10Traffic, 10SRE: CDN doesn't validate request-target - https://phabricator.wikimedia.org/T318676 (10Vgutierrez) 05Open→03Resolved [08:46:56] (HAProxyEdgeTrafficDrop) firing: 59% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:56:56] (HAProxyEdgeTrafficDrop) resolved: 69% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [10:02:26] 10Traffic, 10SRE, 10Patch-For-Review: ATS cache read p999 metrics shows up requests taking up to 1 second on cache read operations - https://phabricator.wikimedia.org/T317748 (10Vgutierrez) p:05Triage→03Medium This seems to happen every nine minutes both for upload and text nodes: ` vgutierrez@cp6016:~$... [10:25:49] 10Traffic, 10SRE, 10Upstream: ATS cache read p999 metrics shows up requests taking up to 1 second on cache read operations - https://phabricator.wikimedia.org/T317748 (10Vgutierrez) Reported to upstream in https://github.com/apache/trafficserver/issues/9118 [12:44:55] 10Traffic, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-ulsfo: add HBA355i support to installer - https://phabricator.wikimedia.org/T319067 (10MoritzMuehlenhoff) >>! In T319067#8280847, @BBlack wrote: > I've also found some other breadcrumbs. Runtime buster + 5.10 support is puppetized in `modul... [12:58:01] 10Traffic, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-ulsfo: add HBA355i support to installer - https://phabricator.wikimedia.org/T319067 (10MoritzMuehlenhoff) >>! In T319067#8276581, @BBlack wrote: > The question is why the Debian installer didn't load this automagically, and how we fix that s... [13:41:23] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: Q1:rack/setup/install ulsfo misc class hosts - https://phabricator.wikimedia.org/T317247 (10ayounsi) From diffscan: ` STATUS HOST PORT PROTO OPREV CPREV DNS OPEN 198.35.26.7 22 tcp 0 6 dns4003.wikimedia.org ` That host is exposed to the world without properly config... [13:41:43] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade management routers and switches to Junos 21 - https://phabricator.wikimedia.org/T316529 (10Papaul) @ayounsi @cmooney i am having space issue on msw1-codfw which is preventing me to copy the Junos image to /var/tmp. request system storage cleanup didn't... [14:09:42] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade management routers and switches to Junos 21 - https://phabricator.wikimedia.org/T316529 (10Papaul) This issue was the file was copied first to /tmp before /var/tmp according to @ayounsi so copy the file first to local laptop and use scp to copy the file... [14:23:01] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox: Netbox: use FHRP Groups feature - https://phabricator.wikimedia.org/T311218 (10ayounsi) Ran the following, then confirmed that there is no diff after a Homer run. `lang=python,lines=20 import uuid request_id = uuid.uuid4() user = User.objects.get(us... [14:42:53] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox: Netbox: use FHRP Groups feature - https://phabricator.wikimedia.org/T311218 (10ayounsi) [14:43:36] 10netops, 10Infrastructure-Foundations, 10SRE: Standardize VRRP group IDs - https://phabricator.wikimedia.org/T260363 (10ayounsi) [14:43:43] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox: Netbox: use FHRP Groups feature - https://phabricator.wikimedia.org/T311218 (10ayounsi) a:03ayounsi [14:43:51] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox: Netbox: use FHRP Groups feature - https://phabricator.wikimedia.org/T311218 (10ayounsi) 05Open→03Resolved [14:59:08] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade management routers and switches to Junos 21 - https://phabricator.wikimedia.org/T316529 (10Papaul) [15:08:39] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: Q1:rack/setup/install ulsfo misc class hosts - https://phabricator.wikimedia.org/T317247 (10RobH) >>! In T317247#8283057, @ayounsi wrote: > From diffscan: > ` > STATUS HOST PORT PROTO OPREV CPREV DNS > OPEN 198.35.26.7 22 tcp 0 6 dns4003.wikimedia.org > ` > That hos... [15:09:06] 10Domains, 10SRE, 10Traffic-Icebox: wikibase.org should redirect to wikiba.se - https://phabricator.wikimedia.org/T254957 (10BCornwall) Untagging the Traffic team: While we're happy to help out when this is needed, this currently appears to be more of a discussion with other teams since we are unable by poli... [15:09:15] 10Domains, 10SRE: wikibase.org should redirect to wikiba.se - https://phabricator.wikimedia.org/T254957 (10BCornwall) [15:43:02] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: Q1:rack/setup/install ulsfo misc class hosts - https://phabricator.wikimedia.org/T317247 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin2002 for host dns4003.wikimedia.org with OS bullseye [16:21:30] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: Q1:rack/setup/install ulsfo misc class hosts - https://phabricator.wikimedia.org/T317247 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by robh@cumin2002 for host dns4003.wikimedia.org with OS bullseye completed: - dns4003 (**FAIL**) - Removed... [16:21:40] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: Q1:rack/setup/install ulsfo misc class hosts - https://phabricator.wikimedia.org/T317247 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by robh@cumin2002 for host dns4003.wikimedia.org with OS bullseye executed with errors: - dns4003 (**FAIL**)... [16:24:45] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: Q1:rack/setup/install ulsfo misc class hosts - https://phabricator.wikimedia.org/T317247 (10RobH) So the failure is just for the script results, and its refusing proxy connection to that url, which has since started to work. All items were processed, dns4003 is rea... [16:25:01] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: Q1:rack/setup/install ulsfo misc class hosts - https://phabricator.wikimedia.org/T317247 (10RobH) [17:19:16] 10Traffic: Consider adding X-Analytics subfield for 'has a session cookie' - https://phabricator.wikimedia.org/T319324 (10CDanis) [17:19:58] 10Traffic: Consider adding X-Analytics subfield for 'has a session cookie' - https://phabricator.wikimedia.org/T319324 (10CDanis) [17:22:48] bblack: vgutierrez: any thoughts positive or negative appreciated re ^, happy to write the patch if you like it [17:40:22] cdanis: seems reasonable to me! [17:40:38] thanks :) [17:51:47] 10Traffic, 10SRE, 10Patch-For-Review: per-backend-service concurrency limits in ATS-BE - https://phabricator.wikimedia.org/T306223 (10ssingh) [17:51:59] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Package and deploy ATS 9.1.3 - https://phabricator.wikimedia.org/T309651 (10ssingh) [17:52:29] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Package and deploy ATS 9.1.3 - https://phabricator.wikimedia.org/T309651 (10ssingh) 05Open→03Resolved ` sukhe@cumin2002:~$ sudo cumin 'A:cp' '/usr/bin/traffic_server --version' 92 hosts will be targeted: cp[2027-2042].codfw.wmnet,cp[6001... [17:54:14] ^ the cp hosts upgrade to ATS9 that Traffic was working on is now completed. if you seem something amiss, please let us know [17:55:25] \o/ [17:55:30] congrats sukhe [17:57:02] cdanis: thanks! credit to vgutierrez without whom this wouldn't have been possible :P [17:57:18] congrats to vgutierrez as well :D [18:30:02] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo, 10Patch-For-Review: Q1:rack/setup/install ulsfo misc class hosts - https://phabricator.wikimedia.org/T317247 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host dns4003.wikimedia.org with OS buster [18:57:28] Cdanis: you shouldn't trust sukhe.. he is too humble [18:57:46] I mean I know that, but I still wanted to be polite to him [18:57:50] He did the heavy lifting.. I just tuned some knobs here and there [19:54:56] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo, 10Patch-For-Review: Q1:rack/setup/install ulsfo misc class hosts - https://phabricator.wikimedia.org/T317247 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host dns4003.wikimedia.org with OS buster executed with errors:... [21:25:57] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo, 10Patch-For-Review: Q1:rack/setup/install ulsfo misc class hosts - https://phabricator.wikimedia.org/T317247 (10BBlack) {F35546970}