[02:44:21] 10netops, 06Infrastructure-Foundations, 10observability, 06SRE: LibreNMS reporting no routes learnt from doh/durum Anycast peers at various POPs - https://phabricator.wikimedia.org/T384258#10502803 (10andrea.denisse) Hi @cmooney, I was reviewing the [[ https://github.com/librenms/librenms/releases/tag/25.... [03:18:37] 10netops, 06Infrastructure-Foundations, 10observability, 06SRE, 10SRE Observability (FY2024/2025-Q3): LibreNMS reporting no routes learnt from doh/durum Anycast peers at various POPs - https://phabricator.wikimedia.org/T384258#10502812 (10andrea.denisse) 05Open→03Resolved After upgrading to v25.1... [06:36:06] 06Traffic, 06collaboration-services, 06Language and Product Localization, 10MinT, 13Patch-For-Review: MinT: Fails to download models/files from peopleweb.discovery.wmnet - https://phabricator.wikimedia.org/T383750#10502911 (10KartikMistry) @ABran-WMF Did we get any useful logs? [07:35:17] 06Traffic, 06collaboration-services, 06Language and Product Localization, 10MinT, 13Patch-For-Review: MinT: Fails to download models/files from peopleweb.discovery.wmnet - https://phabricator.wikimedia.org/T383750#10503007 (10ABran-WMF) >>! In T383750#10502911, @KartikMistry wrote: > > @ABran-WMF Did we... [08:30:27] topranks: I've depooled lvs4009 during 60s to test lvs4010 [08:41:15] 06Traffic, 13Patch-For-Review: Remove RSA certificates from puppet - https://phabricator.wikimedia.org/T375569#10503071 (10Vgutierrez) 05In progress→03Resolved nice work @BCornwall, could you open a task for lists1004 exim dual-stack (RSA+ECDSA) configuration? Thanks! [08:45:44] 06Traffic, 06Data-Engineering, 13Patch-For-Review: Rollout haproxykafka on all hosts - https://phabricator.wikimedia.org/T378578#10503087 (10Fabfur) 05In progress→03Resolved [08:53:26] 06Traffic, 06collaboration-services, 06Language and Product Localization, 10MinT, 13Patch-For-Review: MinT: Fails to download models/files from peopleweb.discovery.wmnet - https://phabricator.wikimedia.org/T383750#10503104 (10KartikMistry) a:03KartikMistry [08:54:15] 06Traffic, 06collaboration-services, 10MinT, 10LPL Essential (LPL Essential 2024 Nov-Jan), 13Patch-For-Review: MinT: Fails to download models/files from peopleweb.discovery.wmnet - https://phabricator.wikimedia.org/T383750#10503108 (10KartikMistry) [08:54:31] 06Traffic, 06collaboration-services, 10MinT, 10LPL Essential (LPL Essential 2024 Nov-Jan), 13Patch-For-Review: MinT: Fails to download models/files from peopleweb.discovery.wmnet - https://phabricator.wikimedia.org/T383750#10503110 (10KartikMistry) 05Open→03In progress [09:07:32] 06Traffic: missing ipip-multiqueue-optimizer prometheus metrics for liberica cluster - https://phabricator.wikimedia.org/T385001 (10Vgutierrez) 03NEW [09:08:40] 06Traffic: missing ipip-multiqueue-optimizer prometheus metrics for liberica cluster - https://phabricator.wikimedia.org/T385001#10503144 (10Vgutierrez) p:05Triage→03Medium [09:39:32] 10netops, 06Infrastructure-Foundations, 10observability, 06SRE, 10SRE Observability (FY2024/2025-Q3): LibreNMS reporting no routes learnt from doh/durum Anycast peers at various POPs - https://phabricator.wikimedia.org/T384258#10503188 (10cmooney) >>! In T384258#10502812, @andrea.denisse wrote: > Aft... [09:46:25] 06Traffic, 06collaboration-services, 10MinT, 10LPL Essential (LPL Essential 2024 Nov-Jan), 13Patch-For-Review: MinT: Fails to download models/files from peopleweb.discovery.wmnet - https://phabricator.wikimedia.org/T383750#10503233 (10LSobanski) >>! In T383750#10503102, @gerritbot wrote: > Change #111377... [10:03:30] 06Traffic, 13Patch-For-Review: missing ipip-multiqueue-optimizer prometheus metrics for liberica cluster - https://phabricator.wikimedia.org/T385001#10503270 (10Vgutierrez) 05Open→03Resolved [10:53:18] 10netops, 06Infrastructure-Foundations, 10ops-magru: Jan 2025 - Magru core router connectivity blips - https://phabricator.wikimedia.org/T384774#10503562 (10cmooney) 05Open→03Resolved Gonna close this one, all is stable after ~24h. [11:35:40] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Productionize gnmic network telemetry pipeline - https://phabricator.wikimedia.org/T369384#10503694 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=36d26c8a-4d30-4345-8682-54b6b4882e38) set by cmooney@cumin1002 for 3:00:... [12:11:38] 06Traffic, 10Citoid, 06Editing-team, 10RESTBase Sunsetting, and 2 others: Switchover plan from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10503792 (10Mvolz) >>! In T361576#10488265, @gerritbot wrote: > Change #1113458 **merged** by jenkins-bot: > %%%[mediawiki/services/c... [13:12:13] 06Traffic, 10Phabricator, 06SRE: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228#10503972 (10Aklapper) Does anyone have sufficient understanding to outline the next potential steps in the blurry territories between undermaintained Phorge upstr... [13:18:54] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Productionize gnmic network telemetry pipeline - https://phabricator.wikimedia.org/T369384#10504014 (10cmooney) Moving to //event-value-tag-v2// has been pushed out to all our Netflow VMs and we've seen a nice reduction in CPU usage, plus a... [14:32:55] 06Traffic, 10Maps, 06SRE, 13Patch-For-Review: Allow Wikimedia Maps usage on schoolwiki.in - https://phabricator.wikimedia.org/T383210#10504478 (10ssingh) @MSantos: Hi! This is pending your approval but otherwise is a simple patch to merge. [15:11:31] 10netops, 06Infrastructure-Foundations, 10observability, 10Prod-Kubernetes, and 3 others: Prevent BGP alerts triggering when K8s host maintenance is being done - https://phabricator.wikimedia.org/T384731#10504843 (10lmata) [15:37:58] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: Dec 2024: cr3-ulsfo errors on et-0/0/0 link from cr4 - https://phabricator.wikimedia.org/T384288#10505023 (10RobH) [16:06:46] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: Dec 2024: cr3-ulsfo errors on et-0/0/0 link from cr4 - https://phabricator.wikimedia.org/T384288#10505148 (10RobH) Picked this back up, it had gotten neglected due to not being assigned to me and not having the ops-ulsfo tag and I shou... [16:06:54] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: Dec 2024: cr3-ulsfo errors on et-0/0/0 link from cr4 - https://phabricator.wikimedia.org/T384288#10505158 (10RobH) a:05cmooney→03RobH [17:00:53] 06Traffic, 13Patch-For-Review: Remove RSA certificates from puppet - https://phabricator.wikimedia.org/T375569#10505415 (10BCornwall) [17:48:27] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: Dec 2024: cr3-ulsfo errors on et-0/0/0 link from cr4 - https://phabricator.wikimedia.org/T384288#10505646 (10RobH) Remote hands 01020815 scheduled for 2025-02-04 @ 0800 Pacific (1600 GMT). [18:26:26] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Check link from msw1-eqiad et-0/1/0 to msw2-eqiad et-0/1/0 - https://phabricator.wikimedia.org/T384708#10505845 (10Papaul) Replaced the optic on the msw2 side [19:39:12] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Check link from msw1-eqiad et-0/1/0 to msw2-eqiad et-0/1/0 - https://phabricator.wikimedia.org/T384708#10506123 (10cmooney) >>! In T384708#10505845, @Papaul wrote: > Replaced the optic on the msw2 side Cool, looks ok so far but will... [20:36:02] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Extend sre.network.configure-switch-interfaces cookbook to add sflow and qos config - https://phabricator.wikimedia.org/T379549#10506260 (10cmooney) The above patch I believe will do what we need. Needs some testing I will work with dc-ops... [20:58:36] 10Acme-chief: Drop the CN from Let's Encrypt CSRs - https://phabricator.wikimedia.org/T385104 (10BCornwall) 03NEW [22:55:29] 10Wikimedia-Apache-configuration, 06serviceops, 13Patch-For-Review: Investigate restricting match pattern on /wiki RewriteRule - https://phabricator.wikimedia.org/T357595#10506787 (10RLazarus) 05Open→03Resolved [23:30:20] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Extend sre.network.configure-switch-interfaces cookbook to add sflow and qos config - https://phabricator.wikimedia.org/T379549#10506853 (10cmooney) As a test I ran this for an existing host that had been configured with the current live co...