[07:27:05] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) Rows E/F network racking task - https://phabricator.wikimedia.org/T292095 (10ayounsi) Let's use a new task for the new racks and keep this one for the spines. Speaking of spines we might want to hold on cabling the ne... [08:55:15] vgutierrez, bblack, is https://phabricator.wikimedia.org/T154801 still a valid task? [08:55:29] * vgutierrez looking [08:55:36] thanks! [08:57:41] XioNoX: took the liberty of closing it [08:58:01] awesome! [10:28:39] 10Traffic, 10SRE-Sprint-Week-Sustainability-March2023, 10envoy, 10serviceops, and 2 others: Raw "upstream connect error or disconnect/reset before headers. reset reason: overflow" error message shown to users during outage - https://phabricator.wikimedia.org/T287983 (10Joe) a:03Joe [10:40:06] 10Traffic, 10SRE: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10MoritzMuehlenhoff) During Sprint week I tried to evaluate a setup where we keep Pybal on Python 2 (as shipped in Bullseye) and build the Twisted packages (which no longer ship Py2 packages in Bullseye) (plus the... [13:09:50] 10netops, 10Infrastructure-Foundations, 10SRE-Sprint-Week-Sustainability-March2023, 10ops-eqiad: eqiad: upgrade row C and D uplinks from 4x10G to 1x40G - https://phabricator.wikimedia.org/T313463 (10ayounsi) [13:28:56] 10netops, 10Infrastructure-Foundations, 10SRE: Avoid sub-optimal routing from CR routers to EVPN destinations - https://phabricator.wikimedia.org/T332781 (10cmooney) p:05Triage→03Medium [13:36:04] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Export routes generated from ARP/ND in EVPN - https://phabricator.wikimedia.org/T329369 (10cmooney) Just a note on this task, related to T332781 If we do have stretched L2 segments across multiple LEAFs, we may wish to also export the /32... [15:07:29] 10Traffic, 10SRE-Sprint-Week-Sustainability-March2023, 10conftool, 10Patch-For-Review, 10Sustainability (Incident Followup): Make it easier to create a new requestctl object - https://phabricator.wikimedia.org/T310009 (10Volans) [15:12:07] 10Traffic, 10SRE-Sprint-Week-Sustainability-March2023, 10conftool, 10Patch-For-Review, 10Sustainability (Incident Followup): Make it easier to create a new requestctl object - https://phabricator.wikimedia.org/T310009 (10Volans) I've sent a small improvement proposal in the above patch, let me know what... [15:23:01] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Avoid sub-optimal routing from CR routers to EVPN destinations - https://phabricator.wikimedia.org/T332781 (10cmooney) [15:23:58] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Avoid sub-optimal routing from CR routers to EVPN destinations - https://phabricator.wikimedia.org/T332781 (10cmooney) [16:14:45] 10Traffic: HAProxy 2.6.10 crashing in the text cluster - https://phabricator.wikimedia.org/T332796 (10Vgutierrez) [16:15:10] 10Traffic: HAProxy 2.6.10 crashing in the text cluster - https://phabricator.wikimedia.org/T332796 (10Vgutierrez) p:05Triage→03High [16:27:00] 10Traffic, 10Upstream: HAProxy 2.6.10 crashing in the text cluster - https://phabricator.wikimedia.org/T332796 (10Vgutierrez) reported to upstream in https://github.com/haproxy/haproxy/issues/2088 [16:44:57] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech: Wikidata seems to still be utilizing insecure HTTP URIs - https://phabricator.wikimedia.org/T331356 (10MisterSynergy) Some remarks: * We should consider these canonical HTTP URIs to be //names// in the first place, which are unique worldwide and issued by the Wiki... [16:59:15] 10Traffic, 10SRE, 10Upstream: HAProxy 2.6.10 crashing in the text cluster - https://phabricator.wikimedia.org/T332796 (10Vgutierrez) Gathering data on esams after downgrading: ` vgutierrez@cumin1001:~$ sudo -i cumin 'A:cp-text_esams' 'apt-cache policy haproxy|grep Installed' 8 hosts will be targeted: cp[3050... [17:10:53] 10netops, 10Infrastructure-Foundations, 10Observability-Alerting, 10SRE: Enable OSPF Icinga check for EVPN based switches - https://phabricator.wikimedia.org/T315053 (10cmooney) 05Open→03Resolved I've merged the patch and the EVPN switches are now being checked by Icinga, all looks healthy. [19:28:18] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech: Wikidata seems to still be utilizing insecure HTTP URIs - https://phabricator.wikimedia.org/T331356 (10Ennomeijers) +1!