[05:49:04] 10netops, 10Infrastructure-Foundations, 10SRE, 10observability, 10Patch-For-Review: Prometheus: ingest SONiC metrics - https://phabricator.wikimedia.org/T335027 (10ayounsi) Read only, there are already some "prometheus-*-expoter" images in https://docker-registry.wikimedia.org/ so it might just be a matt... [08:17:01] 10Traffic, 10Data-Services, 10SRE: 2022-09-04 Scraping from AS714 (Apple) against dumps.wikimedia.org saturating network links - https://phabricator.wikimedia.org/T317001 (10Marostegui) 05Stalled→03Resolved I am going to close it for now, Chris, please reopen if you feel there's still work pending here! [09:52:40] 10Traffic, 10Phabricator, 10Release-Engineering-Team (Seen): Phabricator search times out - https://phabricator.wikimedia.org/T291775 (10Aklapper) [11:46:11] 10netops, 10Infrastructure-Foundations, 10SRE, 10observability: Alertmanager rule for network interface errors? - https://phabricator.wikimedia.org/T335350 (10cmooney) p:05Triage→03Low [11:53:08] 10netops, 10Infrastructure-Foundations, 10SRE, 10observability: Alertmanager rule for network interface errors? - https://phabricator.wikimedia.org/T335350 (10ayounsi) FYI we do alert on those on the network side, see "Inbound interface errors" and "Outbound interface errors" on https://librenms.wikimedia.... [13:48:25] 10Traffic, 10Patch-For-Review: Switch to Maglev hashing ('mh') on LVS hosts - https://phabricator.wikimedia.org/T263797 (10BBlack) I think we need to rewind a step here. We do want `mh`, but we want it for the current public `sh` cases (basically: text and upload ports 80+443), and maybe the other three `sh`... [16:36:08] 10Traffic, 10Patch-For-Review: Switch to Maglev hashing ('mh') on LVS hosts - https://phabricator.wikimedia.org/T263797 (10BCornwall) @bblack: The ticket was literally just the title "switch to maglev hashing (mh) on LVS hosts" and I went with it based on that :) Thanks for the clarification. I'll update the t... [16:40:43] 10Traffic, 10Patch-For-Review: Switch Source Hashing ('sh') scheduling on LVS hosts to Maglev hashing ('mh') - https://phabricator.wikimedia.org/T263797 (10BCornwall) [17:05:34] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10Papaul) Waiting on traffic team to come up with a plan on how to install the new serves. In the pass, what we did was to decommission one server re-use the cables... [20:38:55] 10Traffic, 10SRE-OnFire, 10conftool, 10serviceops, 10Sustainability (Incident Followup): Pybal maintenances break safe-service-restart.py (and thus prevent scap deploys of mediawiki) - https://phabricator.wikimedia.org/T334703 (10BCornwall) @bblack and @cdanis: Could the ticket title/description be updat... [20:41:20] 10Traffic, 10Infrastructure-Foundations, 10SRE: Set NEL `success_fraction: 1.0` on HTTP responses for measurement domains - https://phabricator.wikimedia.org/T334608 (10BCornwall) 05Open→03Resolved Thanks for doing that! [20:41:22] 10Traffic, 10Infrastructure-Foundations, 10SRE: Serve an HTTP response for measurement domains directly from Varnish - https://phabricator.wikimedia.org/T332028 (10BCornwall) [20:42:06] 10netops, 10Infrastructure-Foundations: Adjust routing policy to increase SSH session speed from East Asia to toolforge - https://phabricator.wikimedia.org/T334530 (10BCornwall) [20:53:43] 10Traffic: Replace current L4LB with with Katran-based alternative - https://phabricator.wikimedia.org/T332027 (10BCornwall) p:05Triage→03Medium [20:54:14] 10Traffic: Replace current L4LB with with Katran-based alternative - https://phabricator.wikimedia.org/T332027 (10BCornwall) 05Open→03In progress a:03Vgutierrez [20:56:01] 10Traffic, 10Patch-For-Review: Revisit varnish dynamic backends mechanism - https://phabricator.wikimedia.org/T282880 (10BCornwall) [21:00:13] 10Traffic, 10SRE: Let HAProxy handle port 80 - https://phabricator.wikimedia.org/T323557 (10BCornwall) 05Stalled→03In progress