[00:26:29] <wikibugs>	 06Traffic: Remove RSA certificate support - https://phabricator.wikimedia.org/T370837 (10BCornwall) 03NEW
[00:27:07] <wikibugs>	 06Traffic: Remove RSA certificate support - https://phabricator.wikimedia.org/T370837#10009172 (10BCornwall) p:05Triage→03Medium
[09:47:08] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate codfw servers in rows C & D from legacy ASW to LSW - https://phabricator.wikimedia.org/T370630#10009576 (10cmooney)
[09:50:54] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Netbox automation to move selected hosts from ASW to LSW - https://phabricator.wikimedia.org/T370846 (10cmooney) 03NEW p:05Triage→03Medium
[09:51:02] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate codfw servers in rows C & D from legacy ASW to LSW - https://phabricator.wikimedia.org/T370630#10009596 (10cmooney)
[09:51:02] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Netbox automation to move selected hosts from ASW to LSW - https://phabricator.wikimedia.org/T370846#10009595 (10cmooney)
[10:06:16] <wikibugs>	 10netops, 06DBA, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate codfw row C & D database hosts to new Leaf switches - https://phabricator.wikimedia.org/T370852 (10cmooney) 03NEW p:05Triage→03Medium
[10:06:54] <wikibugs>	 10netops, 06DBA, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate codfw row C & D database hosts to new Leaf switches - https://phabricator.wikimedia.org/T370852#10009700 (10cmooney)
[10:06:55] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate codfw servers in rows C & D from legacy ASW to LSW - https://phabricator.wikimedia.org/T370630#10009701 (10cmooney)
[10:15:51] <wikibugs>	 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE, and 2 others: Spin down api_appserver and appserver clusters - https://phabricator.wikimedia.org/T367949#10009738 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=52c5c269-d4e9-4489-a397-00874b75eb1c) set by cgoubert@cumin1002 for 21 days, 0:0...
[10:40:11] <wikibugs>	 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Update codfw LVS connectivity to support new LSW in rows C & D - https://phabricator.wikimedia.org/T370635#10009886 (10cmooney) The LVS moves are a pre-requisite before we start moving other hosts, so I am going to start pepping the...
[10:52:44] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: lvs2012: Move existing row C & D vlans to primary uplink and add new ones - https://phabricator.wikimedia.org/T370862 (10cmooney) 03NEW p:05Triage→03Medium
[10:52:55] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: lvs2012: Move existing row C & D vlans to primary uplink and add new ones - https://phabricator.wikimedia.org/T370862#10009979 (10cmooney)
[10:52:57] <wikibugs>	 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Update codfw LVS connectivity to support new LSW in rows C & D - https://phabricator.wikimedia.org/T370635#10009980 (10cmooney)
[11:24:18] <wikibugs>	 10netops, 06DBA, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate codfw row C & D database hosts to new Leaf switches - https://phabricator.wikimedia.org/T370852#10010089 (10Volans) Without too much previous experience from past migrations, I think we could tackle it per DB section (aka shard)...
[11:26:33] <wikibugs>	 10netops, 06DBA, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate codfw row C & D database hosts to new Leaf switches - https://phabricator.wikimedia.org/T370852#10010096 (10Ladsgroup) This should have the map: https://fault-tolerance.toolforge.org/map?cluster=s1
[11:42:06] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: lvs2012: Move existing row C & D vlans to primary uplink and add new ones - https://phabricator.wikimedia.org/T370862#10010124 (10cmooney)
[12:07:33] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: lvs2012: Move existing row C & D vlans to primary uplink and add new ones - https://phabricator.wikimedia.org/T370862#10010171 (10cmooney)
[12:58:27] <wikibugs>	 10netops, 06DBA, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Migrate codfw row C & D database hosts to new Leaf switches - https://phabricator.wikimedia.org/T370852#10010285 (10Volans) @Ladsgroup that's looks very useful, I didn't know about it, is it mentioned anywhere? I can't find in wikitech....
[13:12:59] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Upgrade anycast-healthchecker to 0.9.8 (from 0.9.1-1+wmf12u1) - https://phabricator.wikimedia.org/T370068#10010334 (10ssingh) https://grafana.wikimedia.org/goto/8urj7LXIR?orgId=1  {F56642739}  The hypothesis that reducing logging should help the CPU...
[14:45:38] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: lvs2011: Move existing row C & D vlans to primary uplink and add new ones - https://phabricator.wikimedia.org/T370891 (10cmooney) 03NEW p:05Triage→03Medium
[14:50:37] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: lvs2011: Move existing row C & D vlans to primary uplink and add new ones - https://phabricator.wikimedia.org/T370891#10010933 (10cmooney)
[14:51:32] <wikibugs>	 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Update codfw LVS connectivity to support new LSW in rows C & D - https://phabricator.wikimedia.org/T370635#10010941 (10cmooney)
[14:51:35] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: lvs2011: Move existing row C & D vlans to primary uplink and add new ones - https://phabricator.wikimedia.org/T370891#10010940 (10cmooney)
[14:54:13] <Southparkfan>	 Hi! A change (see https://gerrit.wikimedia.org/r/c/operations/homer/public/+/1056192) was pushed yesterday that broke 53/udp for ns0.wikimedia.org and ns1.wikimedia.org briefly. Cause was identified quickly (ns0 and ns1 IPs are part of public1-lvs-XXXXX pools, and this change blocked udp traffic to all public1-lvs-XXXXX IPs), but AuthDNS is not an LVS service - any reason ns0 and ns1 IPs were taken from the LVS pools?
[14:56:14] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: lvs2012: Move existing row C & D vlans to primary uplink and add new ones - https://phabricator.wikimedia.org/T370862#10010948 (10cmooney) a:03cmooney
[14:57:07] <sukhe>	 Southparkfan: in a meeting, reading shortly
[14:57:55] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: lvs2011: Move existing row C & D vlans to primary uplink and add new ones - https://phabricator.wikimedia.org/T370891#10010968 (10cmooney)
[14:58:32] <Southparkfan>	 looking at https://gerrit.wikimedia.org/r/c/operations/dns/+/80340 it isn't a recent thing, and https://github.com/wikimedia/operations-dns/blob/345f37c40d498e7a871a364d3ece711c78e6134a/templates/154.80.208.in-addr.arpa#L48 is another SRE wondering whether public1-lvs was "[the] wrong subnet?"
[14:58:33] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: lvs2012: Move existing row C & D vlans to primary uplink and add new ones - https://phabricator.wikimedia.org/T370862#10010969 (10cmooney)
[14:58:42] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: lvs2012: Move existing row C & D vlans to primary uplink and add new ones - https://phabricator.wikimedia.org/T370862#10010971 (10cmooney)
[14:59:44] <Southparkfan>	 in any case, I understand changing IPs is definitely not a trivial task for AuthDNS
[15:05:18] <sukhe>	 now looking
[15:08:19] <sukhe>	 yeah so 208.80.154.224/27 is what we have for the LVS service IPs and ns0 for example is in that indeed
[15:08:52] <sukhe>	 I am really not sure why that is the case though 
[15:09:08] <sukhe>	 this has some historic context maybe that at least I am not familiar with 
[15:10:15] <sukhe>	 all I see in Netbox for example (the ns0 predates that I think?) is "Keep manual DNS ns0.wikimedia.org"
[15:10:37] <Southparkfan>	 only thing I can think of is the lack of a more generic service "pool" that is not tied to a rack or row
[15:12:51] <sukhe>	 I did find this
[15:12:52] <sukhe>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/75071/1/manifests/site.pp
[15:12:58] <sukhe>	         # Yes, this is an eqiad service IP. This is *not* wrong.
[15:12:58] <sukhe>	         # this is a temporary measure for migrating ns0 to eqiad
[15:14:05] <sukhe>	 so maybe it was a temporary measure for some reason (that I can't find) and then just became permanent? :)
[15:14:23] <sukhe>	 https://bash.toolforge.org/quip/AU7VTzhg6snAnmqnK_pc
[15:14:25] <Southparkfan>	 ah dobson, former host in pmtpa
[15:23:53] <sukhe>	 Southparkfan: I will check with bblac.k to see if he has more context
[15:24:51] <Southparkfan>	 pre-anycast ns2 was 91.198.174.239, used to be part of 91.198.174.232/29, titled "out-of-subnet LVS service IPs" 
[15:26:21] <Southparkfan>	 sukhe: thanks a lot :-)
[15:33:29] <Southparkfan>	 gut feeling is that the "LVS-service-IPs" and "public1-lvs-*" were not purely meant for services backed by LVS, but any kind of services, so more likely a naming issue
[15:34:33] <Southparkfan>	 (and I just learned ns0/ns1 no longer depend on static routes per T348041 - very cool!)
[15:34:35] <stashbot>	 T348041: Remove static routes for ns[01] and replace their announcements with bird - https://phabricator.wikimedia.org/T348041
[15:35:40] <sukhe>	 yeah it was long-time coming! we then also moved the advertisements to confctl which means we can dynamically remove servers from say the ns2 pool (or even ns1) as desired
[15:39:23] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Move lvs2014 uplink to lsw1-d2-codfw, connect to private1-d2-codfw and trunk all vlans on primary. - https://phabricator.wikimedia.org/T370897 (10cmooney) 03NEW p:05Triage→03Medium
[15:40:00] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Move lvs2014 uplink to lsw1-d2-codfw and connect to per-rack vlan - https://phabricator.wikimedia.org/T370897#10011160 (10cmooney)
[15:40:15] <Southparkfan>	 as in: anycast_healthchecker can stop advertising a service IP (ns0/ns1) depending on the pooled/depooled state for that machine in conftool?
[15:40:22] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: lvs2014: move uplink to lsw1-d2-codfw and connect to per-rack vlan - https://phabricator.wikimedia.org/T370897#10011161 (10cmooney)
[16:49:05] <wikibugs>	 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Update codfw LVS connectivity to support new LSW in rows C & D - https://phabricator.wikimedia.org/T370635#10011631 (10cmooney)
[16:49:06] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: lvs2014: move uplink to lsw1-d2-codfw and connect to per-rack vlan - https://phabricator.wikimedia.org/T370897#10011630 (10cmooney)
[16:49:36] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: lvs2014: move uplink to lsw1-d2-codfw and connect to per-rack vlan - https://phabricator.wikimedia.org/T370897#10011653 (10cmooney)
[17:02:09] <wikibugs>	 10Wikimedia-Apache-configuration, 06collaboration-services, 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), and 3 others: Apache 2.4.61 throws a 403 Forbidden for links containing %3F - https://phabricator.wikimedia.org/T370110#10011792 (10hashar) >>! In T370110#10007941, @Dzahn wrote: >>>!...
[17:08:17] <wikibugs>	 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Update codfw LVS connectivity to support new LSW in rows C & D - https://phabricator.wikimedia.org/T370635#10011845 (10cmooney)
[17:26:11] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: lvs2014: move uplink to lsw1-d2-codfw and connect to per-rack vlan - https://phabricator.wikimedia.org/T370897#10011966 (10cmooney)
[17:31:03] <wikibugs>	 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: lvs2014: move uplink to lsw1-d2-codfw and connect to per-rack vlan - https://phabricator.wikimedia.org/T370897#10011969 (10cmooney)
[17:31:18] <wikibugs>	 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: lvs2013: move uplink to lsw1-c2-codfw and connect to per-rack vlan - https://phabricator.wikimedia.org/T370927#10011975 (10cmooney)
[18:10:56] <wikibugs>	 06Traffic, 06collaboration-services, 06SRE, 13Patch-For-Review, 10Release-Engineering-Team (Radar): implement anti-abuse features for GitLab (Move GitLab behind the CDN) - https://phabricator.wikimedia.org/T366882#10012178 (10Dzahn) gitlab1003 and gitlab1004 are unchanged.  gitlab2002 now has the new fil...