[01:37:06] 06Traffic, 06DC-Ops, 10ops-codfw, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#10945606 (10Jhancock.wm) @volans give it a shot on cp2044. if you have any issues with it, lmk [02:55:44] 06Traffic: 429 Error from cp5022 when accessing Wikimedia project - https://phabricator.wikimedia.org/T397804 (10Ppfriedrice) 03NEW [06:12:45] 10netops, 06Infrastructure-Foundations, 06SRE: Migrate row E/F network aggregation to dedicated Spine switches - https://phabricator.wikimedia.org/T322937#10945822 (10Morale99) For anyone dealing with fiber connections or testing networks, using a good quality [[ https://www.firefold.com/collections/fibe... [09:05:32] 06Traffic, 10Liberica: Liberica control plane (liberica-cp) should not automatically start on system boot - https://phabricator.wikimedia.org/T396398#10946280 (10Vgutierrez) 05Open→03In progress [12:34:17] 06Traffic, 06DC-Ops, 10ops-codfw, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#10946886 (10Volans) No way, it doesn't work yet, but I need to understand why: `lang=python >>> import xml.etree.ElementTree as ET >>> from xml.dom import minidom >>> sc... [13:45:07] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Alert when anycast-healthchecker withdraws BGP route - https://phabricator.wikimedia.org/T374619#10947176 (10ssingh) I am going to tackle this for the DNS hosts at least and then we can revisit a generic solution. [14:05:06] 06Traffic, 06DC-Ops, 10ops-codfw, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#10947301 (10Volans) By trial and error with Luca we found that the Target parameter wants a list now. Sent new fix. [15:36:08] FYI, we'll be starting on the swift-r[ow] discovery service turndown (T376237) some time after 16:30 UTC. no action requested, just FYI :) [15:36:09] T376237: Turn down unused swift-r[ow] discovery services - https://phabricator.wikimedia.org/T376237 [15:36:14] thanks! [15:36:20] * swfrench-wmf clearly has not broken enough stuff yet today [15:36:49] don't think it was a breakage but also if you are not breaking stuff, you are not working :) (don't quote me) [15:37:54] haha, indeed - more "transient hiccough" than "t-shirt" :) [17:27:24] 06Traffic: Upgrade to ATS 9.2.11 - https://phabricator.wikimedia.org/T397456#10948132 (10CDobbins) [17:43:07] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#10948154 (10VRiley-WMF) Unracked lvs1017 and installing the card now [17:47:05] 06Traffic: Upgrade to ATS 9.2.11 - https://phabricator.wikimedia.org/T397456#10948164 (10BCornwall) [17:49:21] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#10948171 (10VRiley-WMF) [18:08:41] forgot to mention, the swift-r[ow] turndown is done. no notable issues encountered. [18:08:52] cool, thanks! [18:09:08] there is some confd logspam that I'll look into separately [18:10:15] mw-wikifunctions(-ro)? appears to have been turned down in etcd (conftool-data), but not in the service catalog [18:10:24] so the confd resources are still there [18:14:21] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP and NAT configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865 (10Jgreen) 03NEW [18:21:10] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP and NAT configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948245 (10Jgreen) [18:23:49] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP and NAT configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948248 (10Jgreen) [18:30:30] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#10948297 (10VRiley-WMF) [18:32:37] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#10948306 (10VRiley-WMF) Inserted new NIC. Moved the server to the new location (E2, U39, Port 39), ran the netbox script, and everything went through smoothly. @BCornwall it should be ready for the... [19:05:39] 06Traffic, 10MobileFrontend, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: MobileFrontend should declare "X-Subdomain" variance via "Vary" response header - https://phabricator.wikimedia.org/T390929#10948425 (10Krinkle) Summary of chat with `#wikimedia-traffic` on IRC with @ssingh and @Vgutierrez... [19:17:10] 06Traffic: Upgrade to ATS 9.2.11 - https://phabricator.wikimedia.org/T397456#10948446 (10CDobbins) [19:29:27] https://gerrit.wikimedia.org/r/1163856 should clean up the confd logspam [19:51:44] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP and NAT configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948519 (10Jgreen) [19:51:50] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP and NAT configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948521 (10Jgreen) [20:04:30] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP and NAT configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948573 (10Jgreen) [20:05:25] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948575 (10Jgreen) a:05Dwisehaupt→03None [20:05:38] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948578 (10Jgreen) [20:06:10] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948582 (10Jgreen) [20:30:19] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: pfw BGP configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948644 (10Jgreen) [20:37:32] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 13Patch-For-Review: network and DNS configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948654 (10Jgreen) [21:06:57] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 13Patch-For-Review: network and DNS configuration for new eqiad frack pay-lb servers - https://phabricator.wikimedia.org/T397865#10948779 (10Jgreen)