[07:35:51] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Juniper: use export-format state-data json compact - https://phabricator.wikimedia.org/T362523#9833931 (10ayounsi) JTAC was able to confirm/duplicate the bug on 22.3R3-S2.4, they're escalating it to their engineering team. [07:38:15] 07Puppet, 06SRE: Add humorous redirect for fox.wikimedia.org - https://phabricator.wikimedia.org/T352870#9833949 (10SMMpanels) SMM panels are a fantastic method to organise your social media marketing activities and offer practical, affordable solutions. They let companies easily manage several platforms a... [09:45:45] 10CAS-SSO, 10netbox, 06Infrastructure-Foundations, 13Patch-For-Review: Move Netbox authentication to python-social-auth - https://phabricator.wikimedia.org/T308002#9834318 (10ops-monitoring-bot) Deployed netbox to netbox-dev2002.codfw.wmnet with reason: add python-social-auth and update wheels - ayounsi@cu... [11:45:31] 10CAS-SSO, 10netbox, 06Infrastructure-Foundations, 13Patch-For-Review: Move Netbox authentication to python-social-auth - https://phabricator.wikimedia.org/T308002#9834579 (10ops-monitoring-bot) Deployed netbox to netbox-dev2002.codfw.wmnet with reason: add python-jose and update wheels - ayounsi@cumin1002... [11:46:27] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982 (10ABran-WMF) 03NEW [11:48:08] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834595 (10ABran-WMF) [11:49:06] 10netops, 06Infrastructure-Foundations, 06SRE: T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983 (10ABran-WMF) 03NEW [11:50:10] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834613 (10ABran-WMF) [11:50:58] 10netops, 06Infrastructure-Foundations, 06SRE: T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984 (10ABran-WMF) 03NEW [11:51:20] 10CAS-SSO, 10netbox, 06Infrastructure-Foundations, 13Patch-For-Review: Move Netbox authentication to python-social-auth - https://phabricator.wikimedia.org/T308002#9834631 (10ops-monitoring-bot) Deployed netbox to netbox2002.codfw.wmnet,netbox1002.eqiad.wmnet with reason: add CasApereo auth and update whee... [11:51:53] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834628 (10ABran-WMF) [11:58:11] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986 (10ABran-WMF) 03NEW [11:58:49] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad - https://phabricator.wikimedia.org/T365987 (10ABran-WMF) 03NEW [11:58:50] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834657 (10ABran-WMF) [11:59:29] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad - https://phabricator.wikimedia.org/T365988 (10ABran-WMF) 03NEW [11:59:58] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad - https://phabricator.wikimedia.org/T365988#9834682 (10ABran-WMF) [12:01:28] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834684 (10ABran-WMF) [12:02:52] 10netbox, 06Infrastructure-Foundations: Test Netbox-More-Metrics plugin on Netbox 4.0 - https://phabricator.wikimedia.org/T365989 (10SLyngshede-WMF) 03NEW [12:03:31] 10netbox, 06Infrastructure-Foundations: Test Netbox-More-Metrics plugin on Netbox 4.0 - https://phabricator.wikimedia.org/T365989#9834702 (10SLyngshede-WMF) p:05Triage→03Low [12:04:26] 10netbox, 06Infrastructure-Foundations: Test Netbox-More-Metrics plugin on Netbox 4.0 - https://phabricator.wikimedia.org/T365989#9834706 (10SLyngshede-WMF) [12:04:31] 10netbox, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade Netbox to 4.x - https://phabricator.wikimedia.org/T336275#9834707 (10SLyngshede-WMF) [12:04:41] 10netbox, 06Infrastructure-Foundations: Test Netbox-More-Metrics plugin on Netbox 4.0 - https://phabricator.wikimedia.org/T365989#9834709 (10SLyngshede-WMF) [12:04:43] 10netbox, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade Netbox to 4.x - https://phabricator.wikimedia.org/T336275#9834710 (10SLyngshede-WMF) [13:01:55] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9834830 (10ABran-WMF) [13:02:05] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9834831 (10ABran-WMF) [13:03:40] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad - https://phabricator.wikimedia.org/T365988#9834834 (10ABran-WMF) [13:04:12] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad - https://phabricator.wikimedia.org/T365987#9834836 (10ABran-WMF) [13:05:40] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986#9834838 (10ABran-WMF) [13:05:44] 10netops, 06Infrastructure-Foundations, 06SRE: T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9834840 (10ABran-WMF) [13:06:32] 10netops, 06Infrastructure-Foundations, 06SRE: T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983#9834842 (10ABran-WMF) [13:07:38] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad - https://phabricator.wikimedia.org/T365993 (10ABran-WMF) 03NEW [13:08:04] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad - https://phabricator.wikimedia.org/T365993#9834861 (10ABran-WMF) [13:09:11] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad - https://phabricator.wikimedia.org/T365994 (10ABran-WMF) 03NEW [13:09:27] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad - https://phabricator.wikimedia.org/T365994#9834874 (10ABran-WMF) [13:11:04] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995 (10ABran-WMF) 03NEW [13:11:33] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9834886 (10ABran-WMF) [13:13:14] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834888 (10ABran-WMF) [13:16:03] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983#9834899 (10ABran-WMF) [13:16:13] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9834905 (10ABran-WMF) [13:16:40] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad - https://phabricator.wikimedia.org/T365988#9834911 (10ABran-WMF) a:05cmooney→03MatthewVernon [13:16:57] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad - https://phabricator.wikimedia.org/T365987#9834908 (10ABran-WMF) a:05cmooney→03ABran-WMF [13:19:37] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad - https://phabricator.wikimedia.org/T365996 (10ABran-WMF) 03NEW [13:21:15] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad - https://phabricator.wikimedia.org/T365997 (10ABran-WMF) 03NEW [13:21:44] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad - https://phabricator.wikimedia.org/T365997#9834941 (10ABran-WMF) [13:22:35] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad - https://phabricator.wikimedia.org/T365998 (10ABran-WMF) 03NEW [13:22:48] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad - https://phabricator.wikimedia.org/T365998#9834959 (10ABran-WMF) [13:24:08] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834960 (10ABran-WMF) [14:27:11] 10CFSSL-PKI, 06Infrastructure-Foundations: Establish a process to periodically upgrade the CFSSL infrastructure - https://phabricator.wikimedia.org/T365361#9835123 (10joanna_borun) p:05Triage→03Medium [14:27:32] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Arelion IPv6 transit renumbering - https://phabricator.wikimedia.org/T365697#9835124 (10ayounsi) p:05Triage→03Medium a:03ayounsi [14:27:33] 10CFSSL-PKI, 06Infrastructure-Foundations: Alert and automate the renewal of CFSSL intermediate CAs - https://phabricator.wikimedia.org/T365362#9835126 (10joanna_borun) p:05Triage→03Medium [14:47:38] 10SRE-tools, 06Infrastructure-Foundations, 10Puppet-Core, 10Spicerack, 06SRE: Add a cookbook to safely deploy puppet changes - https://phabricator.wikimedia.org/T341442#9835228 (10elukey) [15:01:27] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9835264 (10ABran-WMF) [15:01:36] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983#9835266 (10ABran-WMF) [15:01:49] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9835267 (10ABran-WMF) [15:01:58] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986#9835268 (10ABran-WMF) [15:02:07] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad - https://phabricator.wikimedia.org/T365987#9835269 (10ABran-WMF) [15:02:17] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad - https://phabricator.wikimedia.org/T365988#9835270 (10ABran-WMF) [15:02:25] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad - https://phabricator.wikimedia.org/T365993#9835271 (10ABran-WMF) [15:02:34] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad - https://phabricator.wikimedia.org/T365994#9835272 (10ABran-WMF) [15:02:55] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9835273 (10ABran-WMF) [15:03:02] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad - https://phabricator.wikimedia.org/T365996#9835274 (10ABran-WMF) [15:03:16] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad - https://phabricator.wikimedia.org/T365997#9835275 (10ABran-WMF) [15:03:24] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad - https://phabricator.wikimedia.org/T365998#9835276 (10ABran-WMF) [16:05:02] dear foundations [16:05:13] under what circumstances would I end up seeing this here [16:05:28] https://usercontent.irccloud-cdn.com/file/hgRGAMEm/image.png [16:06:08] I am reimaging mc2049, while I had no problem with mc2050 [16:10:35] cc XioNoX ^^ [16:10:49] mmmh maybe this has nothing to do with the move vlan [16:10:55] did something broke reimages today? [16:11:08] mmm wait [16:11:22] effie: was mc2049 moved of vlan by any chance? [16:11:28] May 27 16:10:40 install2004 dhcpd[930906]: DHCPDISCOVER from ac:a0:9d:00 via 10.193.0.1: network 10.193.0.0/16: no free leases [16:11:37] this is not suppose to happen I believe? [16:12:00] that's unrelated and it's normal [16:12:12] hosts not setup plugged in in the mgmt network, those are BMCs [16:12:13] to my knowledge no changes have been there [16:12:21] ok ok cool [16:13:07] but Arzhel had a simuilar problem earlier [16:13:26] with a host that was moved to the new vlan [16:13:34] so we were thiknking about related problems [16:14:19] interesting [16:15:54] unless there is some network maint work happening, no changes on those hosts [16:16:00] just reimaging them to bookworm [16:16:18] did the preesed got downloaded? [16:16:41] https://usercontent.irccloud-cdn.com/file/9QhDqKZ1/image.png [16:18:08] I would say no, I glimpsed link detections etc [16:18:31] and given the previous screen and this one [16:18:41] where should I look ? [16:19:08] 10.192.32.81 - - [27/May/2024:15:36:31 +0000] "GET /autoinstall/preseed.cfg HTTP/1.1" 200 25554 "-" "debian-installer" [16:21:46] this is quite odd [16:23:14] and like that many other files [16:23:18] as expected [16:23:33] also [16:23:34] 10.192.32.81 - - [27/May/2024:16:04:02 +0000] "GET /autoinstall/scripts/early_command.sh HTTP/1.1" 200 2219 "-" "Wget/1.21.3" [16:24:07] so mauybe it's the execution that failed [16:24:10] not the download [16:25:52] I am still puzzled why it is asking for the netmask to begin with [16:26:31] is not getting it from dhcp? [16:27:20] sorry I don't know if anything has changed in the dhcp stuff in the last few days [16:27:32] from the network devices PoV [16:27:34] sigh I don;t know, I will debug it later or tomorrow [16:28:48] the config is there, in teh main dhcp config [16:29:09] option subnet-mask 255.255.252.0; [16:30:11] I have already reimaged ~10 more mc servers last week [16:30:24] so something else is up [16:30:39] volans: I need to run, but I will take a look tomorrow [19:53:55] effie, volans, found the issue [19:54:00] https://github.com/wikimedia/operations-puppet/commit/13ad38e2e105f10a8ac057b026bd851f08c55044 [19:57:15] with the US being on holiday I won't do a revert today, but we can tackle it tomorrow morning EU time [21:29:49] that's the upside of having this moved to YAML; now we can introduce partman-breaking config changes in totally new ways :-) [22:10:08] nice finding! yeah we should add something to CI, like the generation of the preseed file and passing it to some linter...