[00:47:39] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12169791 (10Papaul) [01:47:05] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12169848 (10Papaul) [02:39:37] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12169932 (10Papaul) [03:34:46] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, and 2 others: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12169969 (10Papaul) I didn't disable one of the BGP session from mr1 to cr since we didn't make the connection from asw1-603-eqsin to cr2-cr3 et-0/0/... [08:31:33] 10netops, 06Infrastructure-Foundations: Investigate network devices disk encryption with TPM - https://phabricator.wikimedia.org/T433579 (10ayounsi) 03NEW p:05Triage→03Low [09:15:56] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: Consider removing cable IDs from interface descriptions - https://phabricator.wikimedia.org/T432689#12170482 (10ayounsi) Merged and deploying it. It's going to take a bit of time as it means a different diff per device. [09:16:13] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: Consider removing cable IDs from interface descriptions - https://phabricator.wikimedia.org/T432689#12170484 (10ayounsi) 05Open→03Resolved a:03ayounsi [11:53:04] 10netops, 06Infrastructure-Foundations, 06SRE: Core router upgrades 2026 #2 - https://phabricator.wikimedia.org/T431748#12171023 (10ayounsi) 05Open→03Resolved a:03ayounsi All done here. [11:55:01] 10netops, 06Infrastructure-Foundations, 06SRE: cr2-esams rpd failure after enabling bgp 'graceful-shutdown' (June 2026) - https://phabricator.wikimedia.org/T429386#12171034 (10ayounsi) 05Open→03Resolved Closing this task as all the problematic 23.4R2-S7 have been upgraded. The remaining ones will go... [11:56:33] 10netops, 06Infrastructure-Foundations, 06SRE: Core router upgrades 2026 #2 - https://phabricator.wikimedia.org/T431748#12171036 (10ayounsi) 05Resolved→03Open a:05ayounsi→03None [13:55:13] sukhe: fyi: https://phabricator.wikimedia.org/T431683#12171446 [13:57:10] thanks, meeting, will read [14:02:05] topranks: oh wow, you have a patch ready already!? [14:05:40] 06Traffic, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Support LVS backend servers using nftables - https://phabricator.wikimedia.org/T433601#12171495 (10ssingh) [14:14:56] sukhe: I couldn't risk you guys moving backwards that threat nerd sniped me good :P [14:15:10] :P [14:15:18] btw tested in dcl for every combo of inputs I could think of I think it's good [14:15:20] topranks: for now I am simply moving to the non-nftables hosts but yeah [14:15:42] I didn't dare remove the nftables for the hosts that we set them in! [14:16:00] haha maybe I misunderstood you [14:17:00] tbh this wasn't hard at all, the tricky bit for me was the puppet part but no harm getting some more practice [14:17:04] topranks: nice work on the patch. I will review carefully and also learn from it. [14:17:20] yeah. I think we will need one additional thing, correct me if I am wrong [14:17:24] https://phabricator.wikimedia.org/T433601#12171476 [14:17:28] > adding a Prometheus exporter for exporting MSS values based on nftables [14:28:56] sukhe: yeah you might be right, I seen that stuff and just assumed no changes were needed [14:29:06] but probably need to check what the actual f that thing does and see [14:29:16] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-urldownloader04 in project deployment-prep - https://phabricator.wikimedia.org/T433510#12171609 (10ssingh) Sorry about that -- I had no idea we had urldownloaders in beta and that the above change would break tho... [14:29:57] topranks: can help with that bit if required, unless you really want to take that too :P [14:30:12] nah I'm good [14:30:24] hopefully won't be hard, if we can work out what it's doing [14:30:31] I can't work out where this file is coming from [14:30:32] yeah, I have some idea [14:30:37] https://www.irccloud.com/pastebin/nvoMkU69/ [14:31:00] it's python so should be easy to modify, not like this puppetcode junk :P [14:31:04] :P [14:31:22] speaking very confidently as someone who has not looked at it yet [14:31:32] it's right there in the repo ignore me [14:31:33] I think it should be mostly a copy-paste of the existing code but for nftables [14:31:59] yeah, that's the path for all the other prometheus collector files [14:32:57] ok prometheus::node_lvs_realserver_mss is fine - it operates at the system level it doesn't look at anything in iptables [14:33:52] yeah you are right [14:33:52] def get_mss(host: str, port: int, version: int) -> Optional[int]: [14:34:03] is ferm/nftable agnostic [14:34:15] cool, then your patch is complete then! [14:34:16] for the other one I guess we have two choices [14:34:41] we either change prometheus-ferm-mss.py so that it will run either the nftables or iptables command as required [14:35:10] or we duplicate it and create a prometheus-nftables-mss.py that does largely the same thing, but runs the nftables command [14:35:31] I personally think the former is probably better [14:37:15] process_output in ferm-mss.py looks a bit messy [14:37:32] and not sure how the other bits will do with a conditional on the former [14:37:38] but will look carefully [14:37:59] switching the realservers for urldownloader service to the non-nftables for now and see if the entire thing works [14:38:01] we won't need to touch process_output, just rename it process_iptables_output [14:38:02] and then will come to this again [14:38:15] topranks: yeah I meant mangling everything in one file basically [14:38:29] the problem as I see it is how to reliably find from the python script if iptables or nftables is in use on a given host [14:38:58] the script and your patch is already doing that in a waY? [14:39:11] $firewall_provider essentially [14:39:17] which is set explicitly via hiera [14:39:34] modules/profile/manifests/lvs/realserver/ipip.pp is also making that assumption [14:39:35] that's in puppet, looking at profile::firewall::provider [14:40:03] so that's easy to use if we want to duplicate prometheus::node_ferm_mss with a new prometheus::node_nftables_mss [14:40:26] the thing I don't like about that is we duplicate the python code to do the prometheus stuff [14:40:42] what if you pass it as a conditional to prometheus::node_mss instead [14:40:44] when it seems fairly simple to have the script run either iptables or nftables, and then keep that code the same [14:40:45] firewall_provider [14:41:02] that's an idea [14:41:05] simple [14:41:16] let me see [14:50:40] sukhe@lvs1020:~$ curl localhost:9090/pools/urldownloader_8080 [14:50:40] urldownloader1004.wikimedia.org: enabled/down/not pooled [14:50:41] urldownloader1003.wikimedia.org: enabled/down/pooled [14:50:49] thoroughly confused, let me run tcpdump [14:51:06] curl http://urldownloader1003.wikimedia.org:8080 of course works [14:51:14] these are the non-nftable hosts [14:57:35] it's not quite as good as "outerfaces" but "minutely" is also making me laugh [14:57:36] 'interval' => 'minutely' [14:57:57] sukhe: I'll leave that to you [15:06:25] topranks: :P [15:41:16] sukhe: so actually looking at this script I think it might be better to just have separate files [15:41:35] most of the code relates to the specific iptables output parsing, there isn't a whole lot that would be "shared" between them [15:41:52] 10netops, 06Traffic, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: eqsin downtime scheduling for switch upgrade - https://phabricator.wikimedia.org/T433097#12171900 (10RobH) 05Open→03Resolved Resolution Notes: * Work was completed without any unexpected downtime * The fibers/optics needed ha... [15:59:11] topranks: thanks [15:59:36] makes sense I guess plus you know better since you have looked at it deeply [16:05:58] topranks: sorry for sending you down this rabbit hole <3 [16:10:33] 06Traffic, 06Data-Persistence, 13Patch-For-Review: Move thumbnail caching from upload cluster to text - https://phabricator.wikimedia.org/T427465#12172017 (10Ladsgroup) [19:35:51] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12172713 (10ssingh) `urldownloader[12]00[34].wikimedia.org` are now behind LVS as a low-traffic IPIP service in... [20:02:28] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12172754 (10Krinkle)