[10:05:35] is someone from traffic around ? I merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/1055187 but looks like netbox.wikimedia.org still points to the old servers, I guess I missed a step? [10:06:33] looking [10:09:17] fabfur: o/ so netbox.discovery.wmnet seems updated correctly [10:09:40] ok [10:10:21] is somehow related with the recent alert on -operations? [10:11:28] which one are you referring to? [10:11:31] (to be on the same page) [10:12:02] ` SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1002:9100` [10:12:12] but probably it's another thing [10:12:25] yeah it's unrelated [10:13:19] XioNoX: what is the current issue? netbox.wikimedia.org showing the 3.x UI? [10:13:41] elukey: yeah [10:13:57] ah! [10:13:59] it updated [10:14:53] maybe puppet run on lvs hosts? [10:15:15] this is the part that I am curious, it doesn't have an lvs stanza [10:15:19] why LVS? [10:15:27] that's discovery from ATS directly IIRC [10:15:36] or something like that I dont' recall the details [10:15:42] volans: yes but what does it update the discovery record itself? [10:15:52] maybe puppet ruins on ATS? dunno [10:16:09] it must be puppet on dns nodes though [10:16:22] anyway, it's puppet :) [10:16:31] yes exactly [10:16:35] it is puppet on dns nodes [10:16:42] I ran puppet on cp6* thinking it was there, but it didn't work for me [10:16:51] ok, so it's dns :) [10:16:53] thx! [10:17:23] elukey: what does puppet on dns nodes? I'm clearly missing something [10:17:23] I'll run puppet on dns nodes to make it more consistent [10:18:09] 2024-07-22T10:16:09.504744+00:00 dns1004 puppet-agent[1954665]: (/Stage[main]/Profile::Dns::Auth::Discovery/File[/etc/gdnsd/discovery-geo-resources]/content) - codfw => 10.192.0.96 [10:18:13] 2024-07-22T10:16:09.504795+00:00 dns1004 puppet-agent[1954665]: (/Stage[main]/Profile::Dns::Auth::Discovery/File[/etc/gdnsd/discovery-geo-resources]/content) - eqiad => 10.64.0.186 [10:18:17] 2024-07-22T10:16:09.504846+00:00 dns1004 puppet-agent[1954665]: (/Stage[main]/Profile::Dns::Auth::Discovery/File[/etc/gdnsd/discovery-geo-resources]/content) + codfw => 10.192.0.54 [10:18:21] 2024-07-22T10:16:09.504896+00:00 dns1004 puppet-agent[1954665]: (/Stage[main]/Profile::Dns::Auth::Discovery/File[/etc/gdnsd/discovery-geo-resources]/content) + eqiad => 10.64.0.103 [10:18:25] volans: --^ [10:18:51] ahhhh [10:18:52] got it [10:18:58] yeah TIL as well :) [10:19:10] running puppet on dns* with batch 2 atm [10:19:11] totally forgot about this step [11:54:21] 10netops, 06Infrastructure-Foundations, 06SRE: Set Leaf switches in Codfw rows C & D to active and make new vlans live - https://phabricator.wikimedia.org/T370629 (10cmooney) 03NEW p:05Triage→03Medium [12:11:47] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate codfw servers in rows C & D from legacy ASW to LSW - https://phabricator.wikimedia.org/T370630 (10cmooney) 03NEW p:05Triage→03Medium [12:42:13] 10netops, 06Infrastructure-Foundations, 06SRE: Add data to automation for new switches in codfw C/D - https://phabricator.wikimedia.org/T369106#10002128 (10cmooney) [13:30:38] 06Traffic, 10conftool, 13Patch-For-Review: Allow integrating requestctl rules into haproxy - https://phabricator.wikimedia.org/T369606#10002259 (10Joe) 05Open→03In progress a:03Joe [13:39:30] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Update codfw LVS connectivity to support new LSW in rows C & D - https://phabricator.wikimedia.org/T370635 (10cmooney) 03NEW p:05Triage→03Medium [14:49:16] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Request additional mgmt IP range for frack servers - https://phabricator.wikimedia.org/T370164#10002716 (10joanna_borun) p:05Triage→03High a:03cmooney [16:10:20] 06Traffic, 06Data Products, 06Data-Engineering, 10Observability-Logging: New software: haproxykafka - https://phabricator.wikimedia.org/T370668 (10Fabfur) 03NEW [16:28:22] 06Traffic, 06Data Products, 06Data-Engineering, 10Observability-Logging: New software: haproxykafka - https://phabricator.wikimedia.org/T370668#10003489 (10Ottomata) I might be out of my league here, but have yall considered the [[ https://www.haproxy.com/blog/extending-haproxy-with-the-stream-processing-o... [16:32:33] 10Wikimedia-Apache-configuration, 06collaboration-services, 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), and 3 others: Apache 2.4.61 throws a 403 Forbidden for links containing %3F - https://phabricator.wikimedia.org/T370110#10003530 (10sbassett) [16:32:48] 10Wikimedia-Apache-configuration, 06collaboration-services, 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), and 3 others: Apache 2.4.61 throws a 403 Forbidden for links containing %3F - https://phabricator.wikimedia.org/T370110#10003536 (10sbassett) [16:36:06] 10Wikimedia-Apache-configuration, 06collaboration-services, 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), and 3 others: Apache 2.4.61 throws a 403 Forbidden for links containing %3F - https://phabricator.wikimedia.org/T370110#10003546 (10Mstyles) @Aklapper since the gerrit patch is public... [16:36:21] 06Traffic, 06Data Products, 06Data-Engineering, 10Observability-Logging: New software: haproxykafka - https://phabricator.wikimedia.org/T370668#10003563 (10Fabfur) [16:41:28] 06Traffic, 06Data Products, 06Data-Engineering, 10Observability-Logging: New software: haproxykafka - https://phabricator.wikimedia.org/T370668#10003577 (10Vgutierrez) https://github.com/negasus/haproxy-spoe-go could be handy if we go down that road and we don't want to get dirty writing C code :) [17:51:28] 06Traffic, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Replace or remove Debian Buster VMs in 'traffic' cloud-vps project - https://phabricator.wikimedia.org/T360710#10004020 (10Andrew) Mind if I delete traffic-puppetmaster-buster.traffic.eqiad1.wikimedia.cloud? [17:53:09] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: Update codfw LVS connectivity to support new LSW in rows C & D - https://phabricator.wikimedia.org/T370635#10004023 (10cmooney) [17:53:11] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Migrate codfw servers in rows C & D from legacy ASW to LSW - https://phabricator.wikimedia.org/T370630#10004024 (10cmooney) [21:11:57] 10Wikimedia-Apache-configuration, 06collaboration-services, 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), and 3 others: Apache 2.4.61 throws a 403 Forbidden for links containing %3F - https://phabricator.wikimedia.org/T370110#10005017 (10Aklapper) Upstreamed as https://we.phorge.it/T15889 [21:26:06] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, and 2 others: Request additional mgmt IP range for frack servers - https://phabricator.wikimedia.org/T370164#10005058 (10cmooney) OK I have allocated 10.195.1.0/25 in Netbox and configured 10.195.1.1 as a secondary IP on pfw3-codfw on the mgmt... [22:39:17] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad - https://phabricator.wikimedia.org/T365998#10005223 (10bking) `elastic110[0-2]` are banned and ready , as is `wdqs1016`.