[07:33:40] <RhinosF1>	 The bot to update topic with on call isn't working
[07:53:21] <XioNoX>	 lvl=eror msg="could not find the topic for this channel stored. Is the bot in the channel?"
[07:53:47] <XioNoX>	 !oncall-now
[07:53:47] <sirenbot>	 Oncall now for team SRE, rotation batphone:
[07:53:47] <sirenbot>	 A.mir1, j.ynus, l.sobanski, s.lyngs, l.mata, X.ioNoX, c.danis, m.arostegui, s.ukhe, q.uestion_mark, v.gutierrez, m.oritzm, a.pergos, v.olans, a.rnoldokoth, j.elto, e.ffie, j.bond, _.joe_, k.ormat
[07:54:37] <XioNoX>	 not sure how to force it to update the topic (cc _joe_ )
[07:56:04] <_joe_>	 https://wikitech.wikimedia.org/wiki/Vopsbot#Restart_sirenbot
[07:56:17] <_joe_>	 I'm on strike today against my fascist government, sorry :)
[08:05:04] <XioNoX>	 yep I restarted it already
[08:05:19] <XioNoX>	 _joe_: enjoy! 
[08:05:47] <RhinosF1>	 Thanks XioNoX
[08:49:04] <elukey>	 hello folks!
[08:49:13] <elukey>	 so far changeprop seems behaving in the right way
[08:49:20] <elukey>	 I don't see strange backlogs etc..
[08:49:30] <elukey>	 but if you see anything weird, please ping me
[09:06:57] <vgutierrez>	 moritzm: for IPIP encapulation purposes regarding IPVS and liberica I need to create some ipip interfaces without a defined endpoint but that parameter seems mandatory per interfaces(5), so I guess that the best approach would be defining them with the "manual" method and then provide the proper ifup and ifdown scripts to perform the required configuration?
[09:08:24] <moritzm>	 I think so, yes
[09:08:48] <moritzm>	 the other option would be to pilot systemd-networkd for the new loadbalancers
[09:09:16] <vgutierrez>	 those interfaces are required on both sides, LBs and realservers
[09:09:40] <moritzm>	 ah right, then let's stick with the up/ifdown option :-)
[09:09:40] <vgutierrez>	 and that's a plethora of different instances
[09:10:09] <vgutierrez>	 yep.. I could test systemd-networkd stuff on small instances like ncredir or acme-chief if we are set to go down that road
[09:10:16] <vgutierrez>	 but that's another story :)
[09:12:01] <moritzm>	 yeah, systemd-networkd is definitely on the things to look at for next year, but that's another story indeed
[09:56:01] <klausman>	 jbond: I have run into an oddity with the puppet v7 migration, one of the machines I am updating for some reason can't see the package 'puppet-agent' (https://phabricator.wikimedia.org/P53538)
[09:58:49] <jbond>	 klausman: ack let me take a look which cumin host you using?
[09:58:55] <klausman>	 1001
[09:59:08] <klausman>	 the cookbook is still sitting on the "retry" prompt.
[09:59:11] <jbond>	 ack
[09:59:21] <klausman>	 Thing is, I _think_ it's me using hiera wrong
[10:00:53] <klausman>	 I did ml-etcd1003 earlier, that was fine, but I used a host override in hiera. This one is a site-wide one (hieradata/hosts/ml-etcd1003.yaml vs. hieradata/role/eqiad/etcd/v3/ml_etcd.yaml)
[10:02:33] <jbond>	 klausman: there is nothing in hieradata/role/eqiad/etcd/v3/ml_etcd.yaml are you sure yuo merged?
[10:03:11] <jbond>	 klausman: also curious is there a reaon not to just go ahead and do the hole role with migrate-role?
[10:03:31] <klausman>	 I don't want to affect codfw yet
[10:04:07] <jbond>	 ok
[10:04:11] <klausman>	 I indeed forgot to merge %-)
[10:04:24] <jbond>	 :) thats shold be the right file thuogh
[10:04:26] <klausman>	 Sorry for the noise
[10:04:30] <jbond>	 np
[10:05:06] * jbond has deteched
[10:05:23] <klausman>	 ty. I suppose I should start over the cookbook instead of retrying?
[10:05:34] <jbond>	 klausman: yuo should be able to merge and hit retry
[10:06:07] <taavi>	 hitting retry would only retry the installation, not the puppet run, right?
[10:06:19] <klausman>	 yeah, that was my thinking
[10:06:39] <jbond>	 i think it dose both but let me doubl;e check
[10:07:33] <klausman>	 I ran rpa manually on the machien and retry seems to now work
[10:07:58] <jbond>	 as alwas taavi is right it would have just retried the install :)
[10:08:07] <jbond>	 but yes that also works
[10:08:37] <klausman>	 thanks again, both of you.
[10:09:09] <klausman>	 Great Success \o/
[10:09:27] <jbond>	 np
[11:48:54] <Emperor>	 sigh, it has been 0 days since I left a CR with draft comments and forgot to also hit "reply" to actually send them :(
[11:49:25] <jynus>	 Emperor: happened to me multiple times
[11:55:49] <jbond>	 klausman: taavi: fyi i just sent a patch so if that happens again yuo should be able to hit retry
[11:56:05] <klausman>	 excellent, ty!
[11:56:20] <jbond>	 or shuld i say it will throw and error and bring you back to the update yamle bit
[12:59:12] <hashar>	 hello, I could use a merge to update a tox configuration setting ( whitelist_externals -> allowlist_externals ) https://gerrit.wikimedia.org/r/c/operations/software/+/955880  
[12:59:26] <hashar>	 it is not breaking anything for now, but will when I eventually manage to upgrade `tox`
[13:48:02] <klausman>	 Argh, I think I messed up two hosts. Puppet may have run between me merging the hiera change and me starting the cookbook :-/
[14:16:38] <jbond>	 klausman: yes that will happen you need to make sure that you merge the change while the cookbook is paused as thats the oint where we have ensure puppet wont run and mess things up
[14:16:49] <jbond>	 klausman: let me know the hosts and i can manually fix
[14:17:30] <klausman>	 WOrking on it with Luca atm, will report back if needed
[14:17:40] <jbond>	 ack
[14:25:14] <elukey>	 jbond: o/ ml-serve1001 - One thing that I tried was to clean up the ml-serve1001 cert (and subsequent CSRs) on puppetmaster1001, and cleaned up also /var/lib/puppet/ssl on ml-serve1001, but now when the host regenerates the cert I see come CSR mismatch issues
[14:25:40] <elukey>	 if you have a moment to check the host I'd be grateful, sorry in advance if I messed up more
[14:26:06] <jbond>	 elukey: ack the issue is that its now taking to the puppet7 stuff so you will also need to clear the certs requested there
[14:26:31] <elukey>	 ahhh ok, and "there" is puppetserver1001?
[14:26:47] <jbond>	 elukey: yes im on it though
[14:26:54] <elukey>	 okok not doing anything
[14:30:45] <jbond>	 elukey: puppt is running now, ialso needed to manually update puppet
[14:30:59] <jbond>	 as in apt-get install puppey
[14:31:28] <Emperor>	 🐕
[14:35:49] <elukey>	 ack thanks!
[14:39:53] <jbond>	 np
[14:46:09] <klausman>	 It would be a lot harder for me to be angry at the thing if it indeed was named Puppy
[14:48:34] <jbond>	 hehe
[14:59:59] <claime>	 puppy agent sounds like a kid's spy movie
[15:00:06] <claime>	 Or a knock off paw patrol
[15:04:05] <mutante>	 https://en.wikipedia.org/wiki/Puppy_Linux
[15:11:48] <brouberol>	 it was also the name of the minimalistic datadog agent that was developped for IoT devices & such
[15:13:47] <claime>	 Oh I remember installing Puppy Linux on old old OLD boxes :D
[15:14:25] <claime>	 I didn't know it was still alive-ish
[15:15:54] <Emperor>	 is there a quick&easy way to answer "what racks are [hostnames] in?" I could query netbox for each in turn, but imagine for a moment I am very lazy :)
[15:16:38] <Emperor>	 facter doesn't seem to want to tell me
[15:16:43] <claime>	 Emperor: Cumin them and grep in /var/run/motd.dynamic ?
[15:16:45] <claime>	 :p
[15:16:45] <XioNoX>	 Emperor: with cumin you can probably grep and parse the MOTD :)
[15:16:57] <claime>	 XioNoX: Great minds and all that
[15:17:00] <XioNoX>	 claime: ^5
[15:17:14] <claime>	 XioNoX: ^5
[15:18:08] <Emperor>	 :)
[15:18:30] <Emperor>	 thanks, that is I think the tip I needed
[15:20:15] <Emperor>	 "</var/run/motd.dynamic sed -ne 's/.* and rack //p'"
[15:21:27] <claime>	 tbf it could be added as a fact :D