[13:32:30] does anyone know offhand, is it source package names or binary package names that go into modules/aptrepo/files/updates ? [13:37:02] cdanis: -S is source packages, -P is binary packages [13:37:12] thanks taavi [13:37:21] had just opened man grep-dctrl [13:50:13] for the switchover today, to view tmux on cumin1003: sudo -u blake tmux attach -rt blake [13:50:32] i'll be locking scap in about 10m, and we're targeting the read-only piece of the switchover for 15:00 UTC [13:51:32] bjensen: will coordination happen here or on -operations? [13:51:56] marostegui: let's use operations [13:52:06] ok [15:04:34] XioNoX: topranks: our ulsfo woes seem to be continue I think :( [15:04:43] sukhe@bast4006:~$ ping 10.3.0.10 [15:04:46] 64 bytes from 10.3.0.10: icmp_seq=1 ttl=60 time=172 ms [15:05:01] just prepend another 5 times [15:05:41] sukhe@bast4006:~$ dig @10.3.0.1 CHAOS TXT id.server. +short [15:05:41] "dns4004" [15:05:43] that's fine [15:06:05] so it seems to only affect 10.3.0.10 somehow, let me check I guess if we are advertising it properly [15:08:22] 10.3.0.10/32 10.128.0.20 64612 64605 I [15:09:37] doesn't affect DNS hosts thankfully [15:24:38] yeah so this is the way I would go with it: [15:24:38] https://gerrit.wikimedia.org/r/c/operations/homer/public/+/1260734 [15:24:49] and just worry about as-path-length equalization within a pop [15:24:57] I won't claim to review it so leaving the fun to you and Arzhel :P [15:25:01] it will all be reviewed this year hopefully (confed setup etc) [16:53:35] I'm getting a 429 from config-master.wikimedia.org when running `wmf-update-known-hosts-production` - Is this a known issue? [16:54:55] non compliant UA :D [16:54:58] works for me, do you have the last version? [16:55:25] volans: Oh, I will check [16:55:30] btullis: wmf-laptop package up to date? the fix you need is https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/wmf-laptop/+/a111d61674dbf9605abef183394decfe01bf08ad [16:55:39] I think the last version of the package fixes that yeah [17:00:20] I'm getting the following error msg when committing on the private git repo on puppetserver1001 [17:00:20] ⚠️ Something went wrong! Maybe you attempted to rewrite history? [17:00:28] OK, thanks all. I just installed 1.0.5 over 1.0.1 - It still gives me the same error for now, but that's probably just because I'm already blocked. I'll try again tomorrow. [17:00:37] I was only committing, not rebasing nor amending [17:02:17] actually the real error message is [17:02:17] `fatal: Need to specify how to reconcile divergent branches.` [17:02:27] should I be committing from another host? [17:05:01] and if so, should I reset --hard to before my commit? [17:05:35] I only ever do my commits to private from puppetserver1001 - Never had this error before. [17:07:25] brouberol: hmmm, the commit before yours seems to have been rewritten at some point [17:07:54] are you looking at the reflog? [17:08:17] no, comparing /srv/git/private and /srv/git/operations/private [17:08:30] hmm indeed [17:08:30] c4035354b HEAD@{1}: commit (amend): (jmm) Add keytab for install4004 T418993 [17:08:30] 8110bc6d9 HEAD@{2}: commit: (jmm) Add keytab for install4004 T418993 [17:08:30] T418993: Migrating ulsfo to routed Ganeti - https://phabricator.wikimedia.org/T418993 [17:08:33] (the latter is the actual copy used by the puppet service after a commit) [17:09:58] brouberol: so, basically the fix is to remove the two commits from /srv/git/private, pull in the 'correct' one from /srv/git/operations/private, and then add your commit again [17:10:01] I can do that if you want [17:10:23] moritzm isn't in this chan but you should probably give him a heads up [17:10:44] yes please. I'm confident I could figure it out, but if you _know_ the right way, then yes please [17:10:52] I don't want to mess it up further at 6PM [17:12:55] claime: uhh why not [17:13:39] see that box on https://wikitech.wikimedia.org/wiki/Puppet#Private_puppet [17:14:39] brouberol: please double check the diff currently there and do your commit again [17:15:20] it's now working! Thank you taavi [17:18:48] taavi: why not what? [17:19:38] Moritz is out this week, he'll be back Monday and presumably back in IRC then :) [18:38:17] on-callers: (unless I hear otherwise) I will soon begin rolling reboots of the sessionstore cluster, starting first in codfw, then eqiad. no impact is expected! [18:40:07] urandom: oncallnext here, but sounds good to me - I'll skip sessionstore in my current wikikube rollouts just so we don't conflict, any other services we should coordinate? [18:40:47] rzl: oh, if you have rollouts with potential conflicts, I can waive off for another day [18:41:01] there is no rush [18:41:17] no you're fine, I'm just helmfile applying on every service and I don't want to mess with you while you're working :) [18:41:35] alternatively I'm most of the way done and I can just ping you, shouldn't be more than 20 minutes or so [18:42:10] k, yeah I doubt that would cause any issues, but in the interest of reducing variability 🤷‍♂️ [18:42:23] sure 👍 I'll ping you here [18:42:35] thanks! [18:52:07] urandom: over to you -- envoy is upgraded for sessionstore (and everything else) in codfw but not eqiad, so if you're helmfile applying you'll see a version diff, it's safe to deploy [18:53:16] rzl: awesome; thanks! [19:07:05] inflatador: lvs has puppet disabled: Are you still rolling out new services? [19:07:20] brett: nope, that was most certainly me that forgot, not inflatador [19:07:24] sorry about that -- I am fixing it [19:07:27] oh, sorry for the ping! [19:07:30] No worries [19:07:46] Thanks for getting that active/active turned up, it's already helping big time [19:09:10] inflatador: you did the work :) [19:09:17] brett: all good now, sorry for the noise [19:09:38] some of it, anyway ;) [20:02:58] does anyone know if it's normal for newly-provisioned certs to be backdated? I can't remember this happening in the past. Like I just provisioned `opensearch-ipoid-test.discovery.wmnet` with the Discovery issuer, and it's saying `Not Before: Thu Mar 19 05:33:00 UTC 2026` [20:03:45] the discovery intermediate itself expires `Sun May 03 13:54:00 UTC 2026` so maybe that has something to do with it?