[08:32:55] 10Traffic, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10fgiunchedi) [08:47:48] 10netops, 10Infrastructure-Foundations, 10SRE: Netflow/pmacct: use forwardingStatus - https://phabricator.wikimedia.org/T331707 (10ayounsi) Indeed! looks like Cisco specific :( I sent an email t our account rep just in case: > Additionally I was wondering if Junos supported in any way forwardingStatus in IP... [08:50:02] 10Traffic, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Vgutierrez) [09:28:16] 10Traffic, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row C switches upgrade - https://phabricator.wikimedia.org/T331882 (10elukey) [09:51:19] 10Traffic, 10netops, 10DC-Ops, 10Infrastructure-Foundations, and 2 others: Q4/Q1:knams racking elevations & planning - https://phabricator.wikimedia.org/T331886 (10cmooney) >>! In T331886#8688615, @ayounsi wrote: >> One of my concerns is our other caching sites use matched routers for redundancy and we cou... [10:00:47] 10Traffic, 10DBA, 10Data Pipelines, 10Data-Engineering-Planning, and 9 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10elukey) [12:27:55] 10Traffic, 10SRE: Deploy Wikidough: Experimental DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT) public resolver - https://phabricator.wikimedia.org/T252132 (10Johan) [12:28:10] 10Traffic, 10SRE: Deploy Wikidough: Experimental DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT) public resolver - https://phabricator.wikimedia.org/T252132 (10Johan) [13:33:34] 10Traffic, 10netops, 10DC-Ops, 10Infrastructure-Foundations, and 2 others: Q4/Q1:knams racking elevations & planning - https://phabricator.wikimedia.org/T331886 (10ayounsi) > Unless we want to replace both at that stage? Probably not > Ideally, longer-term, it would be nice to have both racks fairly symme... [13:34:59] 10netops, 10Infrastructure-Foundations, 10SRE: Netflow/pmacct: use forwardingStatus - https://phabricator.wikimedia.org/T331707 (10ayounsi) 05Open→03Declined Closing this task. I'll reopen if there is anything useful that comes out of the conversation. Nothing too interesting for us in pmacct changelog n... [15:21:04] 10Traffic: L4LB tracking task - https://phabricator.wikimedia.org/T332027 (10Vgutierrez) [15:30:06] 10Traffic, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row C switches upgrade - https://phabricator.wikimedia.org/T331882 (10herron) [16:35:35] 10Traffic, 10Sustainability (Incident Followup): cp3050 seemd more affected then otheres in recent incident - https://phabricator.wikimedia.org/T330682 (10akosiaris) Removing #SRE as the more specific team is tagged already. [16:55:11] 10netops, 10Infrastructure-Foundations, 10Sustainability (Incident Followup): Cr1-eqiad comms problem when moving to 40G row D handoff - https://phabricator.wikimedia.org/T320566 (10akosiaris) Removing #SRE, the more specific SRE team is already tagged. [16:57:30] 10Traffic, 10conftool, 10Patch-For-Review, 10Sustainability (Incident Followup): requestctl can't act on cache hits - https://phabricator.wikimedia.org/T317794 (10akosiaris) Removing #SRE, has already been triaged to a more specific SRE subteam [16:57:38] 10Traffic, 10Infrastructure-Foundations, 10Patch-For-Review, 10Sustainability (Incident Followup): Rate limiting for hotlinked images - https://phabricator.wikimedia.org/T317799 (10akosiaris) Removing #SRE, this has been already triaged to 2 different SRE subteams [16:59:08] 10netops, 10Infrastructure-Foundations, 10ops-eqiad, 10Sustainability (Incident Followup): eqiad: upgrade row C and D uplinks from 4x10G to 1x40G - https://phabricator.wikimedia.org/T313463 (10akosiaris) Removing #SRE, has already been triaged to a more specific SRE subteam [17:01:29] 10Traffic, 10SRE-OnFire, 10Sustainability (Incident Followup): (Re) evaluate effectiveness / usefulness of varnish/haproxy traffic drop alerts - https://phabricator.wikimedia.org/T310608 (10akosiaris) Removing #SRE, has already been triaged to a more specific SRE subteam [17:08:58] 10Traffic, 10DNS, 10Traffic-Icebox, 10Sustainability (Incident Followup): Automate DNS depools such that manual commits are not required - https://phabricator.wikimedia.org/T303219 (10akosiaris) Removing #SRE, has already been triaged to a more specific SRE subteam [17:10:49] 10netops, 10Infrastructure-Foundations, 10Sustainability (Incident Followup): Optimise WMF WAN Network Configuration - https://phabricator.wikimedia.org/T297355 (10akosiaris) Removing #SRE, has already been triaged to a more specific SRE subteam [17:17:22] 10Traffic, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Experiment with single backend CDN nodes - https://phabricator.wikimedia.org/T288106 (10akosiaris) Removing #SRE, has already been triaged to a more specific SRE subteam [17:18:24] 10Traffic, 10envoy, 10serviceops, 10Sustainability (Incident Followup): Raw "upstream connect error or disconnect/reset before headers. reset reason: overflow" error message shown to users during outage - https://phabricator.wikimedia.org/T287983 (10akosiaris) Removing #SRE, has already been triaged to a m... [17:23:06] 10Traffic, 10Platform Engineering Roadmap Decision Making, 10serviceops, 10MW-1.35-notes (1.35.0-wmf.35; 2020-06-02), and 2 others: Reduce rate of purges emitted by MediaWiki - https://phabricator.wikimedia.org/T250205 (10akosiaris) Removing #SRE, has already been triaged to a more specific SRE subteam(2 o... [18:05:36] 10Traffic, 10Traffic-Icebox, 10Sustainability (Incident Followup): Puppet doesn't restart ferm on failure - https://phabricator.wikimedia.org/T206951 (10Kappakayala) [18:10:54] 10Traffic, 10PyBal, 10Traffic-Icebox, 10Sustainability (Incident Followup): Pybal should reject a confctl configuration that indicates only one cp-text is pooled - https://phabricator.wikimedia.org/T245060 (10Kappakayala) [18:57:39] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ssingh) [19:01:35] 10Traffic, 10SRE: Deploy Wikidough: Experimental DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT) public resolver - https://phabricator.wikimedia.org/T252132 (10ssingh) [19:02:28] 10Traffic, 10SRE, 10Patch-For-Review: Upgrading Wikidough and durum VMs to bullseye - https://phabricator.wikimedia.org/T305589 (10ssingh) 05Open→03Resolved a:03ssingh Closing this in favour of T321309 where it is being tracked and also given that the Ganeti reimaging cookbook exists which was the prim... [19:02:33] 10Traffic, 10SRE: Deploy Wikidough: DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT) public resolver - https://phabricator.wikimedia.org/T252132 (10ssingh) [19:45:01] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (FY2022/2023-Q3): Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=484494a0-6cb6-44... [20:03:30] 10netops, 10Infrastructure-Foundations, 10observability: BFD Status Check Fails when device is unavailable - https://phabricator.wikimedia.org/T332080 (10Aklapper) [20:28:20] 10Traffic, 10SRE: Cleanup and refactor the dnsrecursor module - https://phabricator.wikimedia.org/T332083 (10ssingh) [20:28:47] 10Traffic, 10SRE: Clean up and refactor the dnsrecursor module - https://phabricator.wikimedia.org/T332083 (10ssingh) [20:29:13] 10Traffic, 10SRE: Clean up and refactor the dnsrecursor module - https://phabricator.wikimedia.org/T332083 (10ssingh) p:05Triage→03Medium