[07:35:51] <wikibugs>	 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Juniper: use export-format state-data json compact - https://phabricator.wikimedia.org/T362523#9833931 (10ayounsi) JTAC was able to confirm/duplicate the bug on 22.3R3-S2.4, they're escalating it to their engineering team.
[07:38:15] <wikibugs>	 07Puppet, 06SRE: Add humorous redirect for fox.wikimedia.org - https://phabricator.wikimedia.org/T352870#9833949 (10SMMpanels) SMM panels are a fantastic method to organise your social media marketing activities and offer practical, affordable solutions. They let companies easily manage several platforms a...
[09:45:45] <wikibugs>	 10CAS-SSO, 10netbox, 06Infrastructure-Foundations, 13Patch-For-Review: Move Netbox authentication to python-social-auth - https://phabricator.wikimedia.org/T308002#9834318 (10ops-monitoring-bot) Deployed netbox to netbox-dev2002.codfw.wmnet with reason: add python-social-auth and update wheels - ayounsi@cu...
[11:45:31] <wikibugs>	 10CAS-SSO, 10netbox, 06Infrastructure-Foundations, 13Patch-For-Review: Move Netbox authentication to python-social-auth - https://phabricator.wikimedia.org/T308002#9834579 (10ops-monitoring-bot) Deployed netbox to netbox-dev2002.codfw.wmnet with reason: add python-jose and update wheels - ayounsi@cumin1002...
[11:46:27] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982 (10ABran-WMF) 03NEW
[11:48:08] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834595 (10ABran-WMF)
[11:49:06] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983 (10ABran-WMF) 03NEW
[11:50:10] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834613 (10ABran-WMF)
[11:50:58] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984 (10ABran-WMF) 03NEW
[11:51:20] <wikibugs>	 10CAS-SSO, 10netbox, 06Infrastructure-Foundations, 13Patch-For-Review: Move Netbox authentication to python-social-auth - https://phabricator.wikimedia.org/T308002#9834631 (10ops-monitoring-bot) Deployed netbox to netbox2002.codfw.wmnet,netbox1002.eqiad.wmnet with reason: add CasApereo auth and update whee...
[11:51:53] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834628 (10ABran-WMF)
[11:58:11] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986 (10ABran-WMF) 03NEW
[11:58:49] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad - https://phabricator.wikimedia.org/T365987 (10ABran-WMF) 03NEW
[11:58:50] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834657 (10ABran-WMF)
[11:59:29] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad - https://phabricator.wikimedia.org/T365988 (10ABran-WMF) 03NEW
[11:59:58] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad - https://phabricator.wikimedia.org/T365988#9834682 (10ABran-WMF)
[12:01:28] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834684 (10ABran-WMF)
[12:02:52] <wikibugs>	 10netbox, 06Infrastructure-Foundations: Test Netbox-More-Metrics plugin on Netbox 4.0 - https://phabricator.wikimedia.org/T365989 (10SLyngshede-WMF) 03NEW
[12:03:31] <wikibugs>	 10netbox, 06Infrastructure-Foundations: Test Netbox-More-Metrics plugin on Netbox 4.0 - https://phabricator.wikimedia.org/T365989#9834702 (10SLyngshede-WMF) p:05Triage→03Low
[12:04:26] <wikibugs>	 10netbox, 06Infrastructure-Foundations: Test Netbox-More-Metrics plugin on Netbox 4.0 - https://phabricator.wikimedia.org/T365989#9834706 (10SLyngshede-WMF)
[12:04:31] <wikibugs>	 10netbox, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade Netbox to 4.x - https://phabricator.wikimedia.org/T336275#9834707 (10SLyngshede-WMF)
[12:04:41] <wikibugs>	 10netbox, 06Infrastructure-Foundations: Test Netbox-More-Metrics plugin on Netbox 4.0 - https://phabricator.wikimedia.org/T365989#9834709 (10SLyngshede-WMF)
[12:04:43] <wikibugs>	 10netbox, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade Netbox to 4.x - https://phabricator.wikimedia.org/T336275#9834710 (10SLyngshede-WMF)
[13:01:55] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9834830 (10ABran-WMF)
[13:02:05] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9834831 (10ABran-WMF)
[13:03:40] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad - https://phabricator.wikimedia.org/T365988#9834834 (10ABran-WMF)
[13:04:12] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad - https://phabricator.wikimedia.org/T365987#9834836 (10ABran-WMF)
[13:05:40] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986#9834838 (10ABran-WMF)
[13:05:44] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9834840 (10ABran-WMF)
[13:06:32] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983#9834842 (10ABran-WMF)
[13:07:38] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad - https://phabricator.wikimedia.org/T365993 (10ABran-WMF) 03NEW
[13:08:04] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad - https://phabricator.wikimedia.org/T365993#9834861 (10ABran-WMF)
[13:09:11] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad - https://phabricator.wikimedia.org/T365994 (10ABran-WMF) 03NEW
[13:09:27] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad - https://phabricator.wikimedia.org/T365994#9834874 (10ABran-WMF)
[13:11:04] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995 (10ABran-WMF) 03NEW
[13:11:33] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9834886 (10ABran-WMF)
[13:13:14] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834888 (10ABran-WMF)
[13:16:03] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad  - https://phabricator.wikimedia.org/T365983#9834899 (10ABran-WMF)
[13:16:13] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad  - https://phabricator.wikimedia.org/T365984#9834905 (10ABran-WMF)
[13:16:40] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad - https://phabricator.wikimedia.org/T365988#9834911 (10ABran-WMF) a:05cmooney→03MatthewVernon
[13:16:57] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad - https://phabricator.wikimedia.org/T365987#9834908 (10ABran-WMF) a:05cmooney→03ABran-WMF
[13:19:37] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad - https://phabricator.wikimedia.org/T365996 (10ABran-WMF) 03NEW
[13:21:15] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad - https://phabricator.wikimedia.org/T365997 (10ABran-WMF) 03NEW
[13:21:44] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad - https://phabricator.wikimedia.org/T365997#9834941 (10ABran-WMF)
[13:22:35] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad - https://phabricator.wikimedia.org/T365998 (10ABran-WMF) 03NEW
[13:22:48] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad - https://phabricator.wikimedia.org/T365998#9834959 (10ABran-WMF)
[13:24:08] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9834960 (10ABran-WMF)
[14:27:11] <wikibugs>	 10CFSSL-PKI, 06Infrastructure-Foundations: Establish a process to periodically upgrade the CFSSL infrastructure - https://phabricator.wikimedia.org/T365361#9835123 (10joanna_borun) p:05Triage→03Medium
[14:27:32] <wikibugs>	 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Arelion IPv6 transit renumbering - https://phabricator.wikimedia.org/T365697#9835124 (10ayounsi) p:05Triage→03Medium a:03ayounsi
[14:27:33] <wikibugs>	 10CFSSL-PKI, 06Infrastructure-Foundations: Alert and automate the renewal of CFSSL intermediate CAs - https://phabricator.wikimedia.org/T365362#9835126 (10joanna_borun) p:05Triage→03Medium
[14:47:38] <wikibugs>	 10SRE-tools, 06Infrastructure-Foundations, 10Puppet-Core, 10Spicerack, 06SRE: Add a cookbook to safely deploy puppet changes - https://phabricator.wikimedia.org/T341442#9835228 (10elukey)
[15:01:27] <wikibugs>	 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f5-eqiad - https://phabricator.wikimedia.org/T365982#9835264 (10ABran-WMF)
[15:01:36] <wikibugs>	 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f6-eqiad - https://phabricator.wikimedia.org/T365983#9835266 (10ABran-WMF)
[15:01:49] <wikibugs>	 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9835267 (10ABran-WMF)
[15:01:58] <wikibugs>	 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e5-eqiad - https://phabricator.wikimedia.org/T365986#9835268 (10ABran-WMF)
[15:02:07] <wikibugs>	 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e6-eqiad - https://phabricator.wikimedia.org/T365987#9835269 (10ABran-WMF)
[15:02:17] <wikibugs>	 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e7-eqiad - https://phabricator.wikimedia.org/T365988#9835270 (10ABran-WMF)
[15:02:25] <wikibugs>	 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad - https://phabricator.wikimedia.org/T365993#9835271 (10ABran-WMF)
[15:02:34] <wikibugs>	 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad - https://phabricator.wikimedia.org/T365994#9835272 (10ABran-WMF)
[15:02:55] <wikibugs>	 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e3-eqiad - https://phabricator.wikimedia.org/T365995#9835273 (10ABran-WMF)
[15:03:02] <wikibugs>	 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f1-eqiad - https://phabricator.wikimedia.org/T365996#9835274 (10ABran-WMF)
[15:03:16] <wikibugs>	 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f2-eqiad - https://phabricator.wikimedia.org/T365997#9835275 (10ABran-WMF)
[15:03:24] <wikibugs>	 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 -lsw1-f3-eqiad - https://phabricator.wikimedia.org/T365998#9835276 (10ABran-WMF)
[16:05:02] <effie>	 dear foundations 
[16:05:13] <effie>	 under what circumstances would I end up seeing this here 
[16:05:28] <effie>	 https://usercontent.irccloud-cdn.com/file/hgRGAMEm/image.png
[16:06:08] <effie>	 I am reimaging mc2049, while I had no problem with mc2050
[16:10:35] <volans>	 cc XioNoX ^^
[16:10:49] <volans>	 mmmh maybe this has nothing to do with the move vlan
[16:10:55] <volans>	 did something broke reimages today?
[16:11:08] <effie>	 mmm wait 
[16:11:22] <volans>	 effie: was mc2049 moved of vlan by any chance?
[16:11:28] <effie>	 May 27 16:10:40 install2004 dhcpd[930906]: DHCPDISCOVER from ac:a0:9d:00<snip> via 10.193.0.1: network 10.193.0.0/16: no free leases
[16:11:37] <effie>	 this is not suppose to happen I believe?
[16:12:00] <volans>	 that's unrelated and it's normal
[16:12:12] <volans>	 hosts not setup plugged in in the mgmt network, those are BMCs
[16:12:13] <effie>	 to my knowledge no changes have been there 
[16:12:21] <effie>	 ok ok cool 
[16:13:07] <volans>	 but Arzhel had a simuilar problem earlier
[16:13:26] <volans>	 with a host that was moved to the new vlan
[16:13:34] <volans>	 so we were thiknking about related problems
[16:14:19] <effie>	 interesting
[16:15:54] <effie>	 unless there is some network maint work happening, no changes on those hosts 
[16:16:00] <effie>	 just reimaging them to bookworm 
[16:16:18] <volans>	 did the preesed got downloaded?
[16:16:41] <effie>	 https://usercontent.irccloud-cdn.com/file/9QhDqKZ1/image.png
[16:18:08] <effie>	 I would say no, I glimpsed link detections etc 
[16:18:31] <effie>	 and given the previous screen and this one 
[16:18:41] <effie>	 where should I look ?
[16:19:08] <volans>	 10.192.32.81 - - [27/May/2024:15:36:31 +0000] "GET /autoinstall/preseed.cfg HTTP/1.1" 200 25554 "-" "debian-installer"
[16:21:46] <effie>	 this is quite odd
[16:23:14] <volans>	 and like that many other files
[16:23:18] <volans>	 as expected
[16:23:33] <volans>	 also
[16:23:34] <volans>	 10.192.32.81 - - [27/May/2024:16:04:02 +0000] "GET /autoinstall/scripts/early_command.sh HTTP/1.1" 200 2219 "-" "Wget/1.21.3"
[16:24:07] <volans>	 so mauybe it's the execution that failed
[16:24:10] <volans>	 not the download
[16:25:52] <effie>	 I am still puzzled why it is asking for the netmask to begin with 
[16:26:31] <volans>	 is not getting it from dhcp?
[16:27:20] <volans>	 sorry I don't know if anything has changed in the dhcp stuff in the last few days
[16:27:32] <volans>	 from the network devices PoV
[16:27:34] <effie>	 sigh I don;t know, I will debug it later or tomorrow
[16:28:48] <volans>	 the config is there, in teh main dhcp config
[16:29:09] <volans>	             option subnet-mask 255.255.252.0;
[16:30:11] <effie>	 I have already reimaged ~10 more mc servers last week 
[16:30:24] <effie>	 so something else is up 
[16:30:39] <effie>	 volans: I need to run, but I will take a look tomorrow
[19:53:55] <XioNoX>	 effie, volans, found the issue
[19:54:00] <XioNoX>	 https://github.com/wikimedia/operations-puppet/commit/13ad38e2e105f10a8ac057b026bd851f08c55044
[19:57:15] <XioNoX>	 with the US being on holiday I won't do a revert today, but we can tackle it tomorrow morning EU time
[21:29:49] <moritzm>	 that's the upside of having this moved to YAML; now we can introduce partman-breaking config changes in totally new ways :-)
[22:10:08] <volans>	 nice finding! yeah we should add something to CI, like the generation of the preseed file and passing it to some linter...