[00:08:29] <swfrench-wmf>	 updates: despite various hiccoughs along the way today, the eqiad main-etcd cluster has been ship-of-theseus'd to bookworm.
[00:08:30] <swfrench-wmf>	 * given the hour, I'm going to hands-off for now, and regroup with Traffic on Thursday to coordinate repooling etcd client traffic there.
[00:08:30] <swfrench-wmf>	 * current status, along with details on how to repool eqiad if there's an urgent need in the meantime (albeit unlikely), are at the top of the task description in [0].
[00:08:30] <swfrench-wmf>	 [0] https://phabricator.wikimedia.org/T428495
[08:41:05] <jynus>	 I was like "how come the decom script has found rests of code referencing backup1003???"
[08:41:25] <jynus>	 it turns out it found ms-backup1003 :-DDD
[13:49:29] <swfrench-wmf>	 FYI, following up on my message yesterday, we will soon be repooling eqiad main-etcd for client traffic. no impact expected, but this will involve restarting a couple of things, etc. as usual.
[14:00:54] <sukhe>	 cdanis: thoughts on https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1313985 ? I would need a +1 for it and for deployment I will find someone in PSI but I wanted to get an SRE approval first
[14:01:11] <sukhe>	 the service is up and running so you can verify that, but wanted to get input on the approach
[14:01:50] <cdanis>	 sukhe: +1 but I think eventually we might want to consider making this traffic go via the service mesh
[14:02:58] <sukhe>	 cdanis: yeah, you are not wrong. I tried to go down that path as well but I figured we have to fix urldownloaders eventually so I will do this for now and someone else can do that bit later™
[14:03:06] <cdanis>	 yeah :)
[14:04:12] <sukhe>	 thanks. we will see how the deployment goes.
[14:05:10] <sukhe>	 I am slightly worried about missing some edge case about this since I don't understand the MW side fully. but that's all I could find references to in the code, so let's see
[14:10:29] <taavi>	 is there a k8s network policy that needs updating for the new service addresses?
[14:11:39] <taavi>	 seems like the current policy is host IPs only https://gerrit.wikimedia.org/g/operations/puppet/+/811682f20dece519b952a59ded93ab5d0d8341a2/modules/profile/manifests/kubernetes/deployment_server/global_config.pp#557
[14:12:17] <Mvolz>	 i would appreciate a heads up any time urldownloader changes are made. we still haven't fixed https://phabricator.wikimedia.org/T381372 so url-downloader issues fail silently still :(
[14:13:35] <cdanis>	 yeah that will need the service IP, sukhe do you want to make the patch? otherwise I can
[14:14:06] <sukhe>	 taavi: cdanis: thanks, I can take care of updating it after this deploy
[14:15:07] <sukhe>	 Mvolz: yeah so in theory, this change should help with that. the failure is still silent though since we are not paging on the urldownloader being down but we can change that
[14:15:47] <sukhe>	 I will go over the ticket in detail later but if it is just > url-downloader failures are detected, then yes, things should improve a bit
[14:16:09] <Mvolz>	 awesome!!
[14:16:51] <Mvolz>	 if it's all possible to page that would be desirable
[14:17:13] <Mvolz>	 citoid is a webscraper so if we can't reach the outside world we drop to 0% success ratio on our slo
[14:58:54] <sukhe>	 seems like PCC has been/is broken for deploy1003 for a while
[14:59:00] <sukhe>	 https://puppet-compiler.wmflabs.org/output/1321987/9180/deploy1003.eqiad.wmnet/change.deploy1003.eqiad.wmnet.err
[14:59:33] <sukhe>	   $res = wmflib::puppetdb_query($pql)
[14:59:33] <sukhe>	   if ($res == undef) {
[14:59:33] <sukhe>	     []
[15:03:38] <swfrench-wmf>	 heads-up: eqiad main-etcd is repooled for client traffic. no further work planned for today. many thanks to c.jd91 for the assistance with restarts :)
[15:03:48] <swfrench-wmf>	 sukhe: hmm ... that's non-ideal
[15:04:31] <Raine>	 oh I think I know what that is
[15:04:34] <Raine>	 maybe
[15:04:54] <Raine>	 I can fix it, I have a commit somewhere that I forgot to push 
[15:05:07] <sukhe>	 thanks
[15:05:16] <sukhe>	 wmflib::puppetdb_query is returning an empty array
[15:05:18] <swfrench-wmf>	 wow, nice! and thank you :)
[15:05:21] <sukhe>	 so the undef never matches I think
[15:05:28] <sukhe>	 but yeah, if you have a patch, I am happy to review
[15:08:55] <brouberol>	 Hi! I'm re-imagine an-test-worker100[12], and it seems that the debian installer runs (I can see it in the install-console), and then they reboot in a loop (still visible in the console)
[15:09:18] <brouberol>	 should I ask here, or in #-sre-foundations? 
[15:09:41] <cdanis>	 brouberol: UEFI?
[15:10:27] <brouberol>	 Hmm I'm not sure tbh
[15:10:51] <brouberol>	 how can I know?
[15:11:30] <Raine>	 https://gerrit.wikimedia.org/r/1321999
[15:11:36] <Raine>	 swfrench-wmf/sukhe ^
[15:11:41] <sukhe>	 thanks Raine, looking
[15:12:12] <cdanis>	 brouberol: https://gerrit.wikimedia.org/g/operations/puppet/+/3c6fb5e4e2fae712de7bcb7e99503491df8e81fe/modules/profile/data/profile/installserver/preseed.yaml#47 looks like it's regular old BIOS
[15:12:49] <swfrench-wmf>	 looking
[15:12:55] <brouberol>	 oh right, no -uefi in the preseed
[15:13:31] <brouberol>	 looking at the install-console, both hosts seem to be unable to boot: https://phabricator.wikimedia.org/P95936
[15:14:11] <cdanis>	 if it did get set back from PXE to disk, then, the next step is making sure that grub got installed on the right physical disks
[15:14:57] <cdanis>	 hm
[15:16:41] <cdanis>	 brouberol: I think you need some more stanzas in your partman recipe
[15:17:37] <cdanis>	 like https://codesearch.wmcloud.org/puppet/?q=grub-installer%2F&files=&excludeFiles=&repos=
[15:18:11] <cdanis>	 see also https://phabricator.wikimedia.org/T215183
[15:19:43] <brouberol>	 I will readily admit I know next to nothing about how this works 
[15:21:36] <brouberol>	 grub-installer/bootdev		string	/dev/sda /dev/sdb
[15:21:37] <brouberol>	 grub-installer/only_debian		boolean false
[15:21:37] <brouberol>	 --
[15:21:37] <brouberol>	 would be enough?
[15:22:05] <sukhe>	 thanks for fixing it Raine!
[15:22:28] <cdanis>	 brouberol: it looked like the an-test-worker recipe has 4 devices? but something like that yeah
[15:23:33] <cdanis>	 https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/3c6fb5e4e2fae712de7bcb7e99503491df8e81fe/modules/install_server/files/autoinstall/partman/custom/reuse-analytics-hadoop-test-worker.cfg
[15:23:52] <brouberol>	 yep, I was looking at lsblk, and indeed we go from sda to sdd
[15:26:15] <brouberol>	 I submitted https://gerrit.wikimedia.org/r/c/operations/puppet/+/1322002
[15:30:07] <sukhe>	 Raine: is it fine to merge your patch?
[15:30:19] <sukhe>	 I wanted to test out PCC on the other one and hence the question
[15:30:32] <sukhe>	 (response not expected at this time, whenver you see this)
[15:37:57] <brouberol>	 cdanis: time isn't on my side atm. Do you think I could trouble you with an expedited patch review? _/|_
[15:38:04] <cdanis>	 please
[15:38:13] <cdanis>	 I could use a distraction right now
[15:38:14] <brouberol>	 (link above)
[15:38:53] <cdanis>	 lgtm, run puppet on install* and try again :)
[15:39:35] <brouberol>	 yep, on it
[15:39:36] <brouberol>	 thank you!
[15:39:44] <cdanis>	 lmk if it doesn't work
[15:40:07] <cdanis>	 but that looks to still be the current pattern, in use in some of the reuse- sibling files in that dir even
[15:40:43] <brouberol>	 what confuses me is that I did reimage an-test-worker1003 successfully yesterday
[15:41:03] <cdanis>	 you won the drive vs BIOS drive mapping lottery
[15:41:06] <brouberol>	 but I've learned to leave indecision and doubt at the door when it comes to preseed
[15:41:10] <cdanis>	 s/drive vs/kernel vs/
[15:41:20] <brouberol>	 and try to matrix jump the building
[15:44:24] <Raine>	  sukhe yeah sorry
[15:44:46] <sukhe>	 please don't be :)
[15:48:47] <sukhe>	 merging
[15:56:41] <brouberol>	 cdanis: "Booting GNU/Linux"
[15:56:55] <brouberol>	 I owe you an unlimited amount of drinks, because I was pressed for time
[15:57:17] <cdanis>	 <3
[15:57:22] <cdanis>	 I wish we weren't still using partman
[15:58:39] <brouberol>	 I asked chatgpt to help me and it said "nah, partman brah... figure it out" /s
[15:59:54] <cdanis>	 I do remember hearing some goss about upcoming improvements to the installer, but I don't remember specifics
[16:02:06] <brouberol>	 what I've heard from b.tullis is that debian has been saying "something better is coming" for 10 years
[16:02:49] <cdanis>	 I mean yeah ofc
[16:03:00] <cdanis>	 but I think I heard it from m.oritz this time
[16:14:18] <brouberol>	 oh! 
[16:14:44] <brouberol>	 (FWIW the 2nd host has booted linux and is now getting its puppet certificate taken care of, so thanks a ton)