[09:21:44] <jnuche>	 hi there, there's another migrateLinksTable script flooding the logs: https://logstash.wikimedia.org/goto/2e4d3bf0098e96590e65e167c002dfbe
[09:21:51] <OLGA>	 my photo   https://www.highcpmgate.com/vdvpyk3j?key=f2b21af2cea51f467a32bdc200d0bff7
[09:21:53] <jnuche>	 is anyone around who could take a look/stop the script?
[09:22:03] <jnuche>	 A.mir1 and a.pergos know about the issue but they have a holiday today
[09:22:22] <OLGA>	 my photo   https://www.highcpmgate.com/vdvpyk3j?key=f2b21af2cea51f467a32bdc200d0bff7
[09:22:27] <marostegui>	 jnuche: I can probably kill it, but I am not sure if it is safe?
[09:22:56] <jnuche>	 marostegui: sadly me neither :)
[09:23:09] <marostegui>	 Ah, I think I know how to get it fixed
[09:24:25] <marostegui>	 jnuche: I think this will fix it https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1026096/
[09:24:31] <marostegui>	 Let me deploy that
[09:43:27] <marostegui>	 jnuche: The errors seem to have stopped after pushing that
[09:47:08] <marostegui>	 mmm not, they are back :(
[09:47:47] <jnuche>	 marostegui: yeah, I was about to say
[09:47:56] <jnuche>	 I can still see them https://logstash.wikimedia.org/goto/a68676cb08727d1e7a2fb4d2e464eda5
[09:48:25] <marostegui>	 yeah, es7 is a new section we are setting up. I believe we've added it everywhere already (it is not in use yet)
[09:49:36] <jnuche>	 maybe the script needs to be restarted to pick up the new config, but probably only Amir knows if it's safe to do that
[09:49:58] <marostegui>	 yeah, I can try to create the tables on the cluster, but I am sure that won't fix it, as the error isn't related to that really
[09:50:24] <marostegui>	 Perhaps that script will reload the config after some time (I believe most of the scripts were migrated to do so some time ago)
[09:51:30] <jnuche>	 let's hope for that, muchas gracias de todos modos :)
[09:51:41] <marostegui>	 de nada :)
[10:26:56] <Emperor>	 Is there a git review rune for "update this branch which I previously pushed with git review -R"? I pushed a branch to gerrit, someone else has uploaded new patchsets, and I'd like to update my local branch. I could just do git review -d patchnumber but that makes a new branch.
[10:29:26] <taavi>	 Emperor: one option is to drop your outdated commit from your local branch and then `git review -x <patch>` to cherry-pick the new version to the current branch
[13:16:01] <Amir1>	 jnuche: marostegui I restarted the script so it could pick up the new section, it should be gone now
[13:16:31] <marostegui>	 Amir1: thank you <3
[13:16:39] <Amir1>	 the reconfigure one works for host depools, new sections is something quite new to this (and rare)
[13:16:57] <marostegui>	 yeah :)
[13:19:13] <jnuche>	 Amir1: thank you!
[13:19:16] <jnuche>	 if we run into this again tomorrow, is it safe to kill the script until you have a chance to take a look?
[13:19:30] <marostegui>	 jnuche: it shouldnt happen tomorrow
[13:19:42] <marostegui>	 It was today (and yesterday) cause we were adding two new sections
[13:20:02] <Amir1>	 jnuche: just restart it
[13:20:44] <jnuche>	 will do, thx again to both
[13:21:08] <jnuche>	 (or more like, will ask someone to do it)
[14:30:03] <sukhe>	 jhathaway: Emperor: fyi since I saw this was just pushed
[14:30:08] <sukhe>	 not sure if it is relevant or not:
[14:30:13] <sukhe>	 Info: Loading facts
[14:30:14] <sukhe>	 Error: Facter: Error while resolving custom fact fact='cephadm', resolution='<anonymous>': No such file or directory @ rb_sysopen - /root/.ssh/id_cephadm.pub
[14:30:39] <Emperor>	 sukhe: I think that resolved on the next puppet run
[14:30:46] <sukhe>	 ok thanks! 
[14:30:54] <jhathaway>	 ah, that is my fault, we need to confine the fact to only ceph nodes
[14:31:13] <Emperor>	 I mean, I have a target node which has now has the result-of-the-fact deployed on it, IYSWIM
[14:31:13] <sukhe>	 yeah I just got it again fwiw. it doesn't affect what I am doing but I thought I should let you know
[14:31:56] <jhathaway>	 I forgot that module facts are global, which is an *interesting* desing decision
[14:32:02] <Emperor>	 jhathaway: are you OK to do that? I suspect you'll be quicker than me at it
[14:32:20] <Emperor>	 global> that is a bit surprising, but puppet remains full of surprises for me :)
[14:33:02] <jhathaway>	 :)
[14:33:11] <jhathaway>	 Emperor: on it
[14:33:24] <Emperor>	 <3
[14:33:25] <sukhe>	 Emperor: you and all of us
[14:47:52] <jhathaway>	 sukhe & Emperor: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1026156
[14:55:13] <Emperor>	 LGTM, but I am too slow :)
[20:37:34] <jhathaway>	 does anyone know what creates the acmechief directories, /var/lib/acme-chief/certs/<FOO>, or are there any docs?
[20:48:48] <inflatador>	 jhathaway looks like the deb pkg, based on `dpkg -L acme-chief | grep var`
[20:50:55] <inflatador>	 jhathaway scratch that, I guess you were asking about the subdirs. Sorry
[20:54:19] <sukhe>	 jhathaway: https://gitlab.wikimedia.org/repos/sre/acme-chief/-/blob/main/acme_chief/acme_chief.py?ref_type=heads#L60
[20:54:35] <sukhe>	 the acme-chief package
[21:02:05] <jhathaway>	 hmm, I added the yaml config based on copy past, but mx-out2001 when it makes a request to acmechief2002 the acme-chief process on acmechief2002 throws an exception saying /var/lib/acme-chief/certs/mx-out doesn't exist
[21:02:27] <jhathaway>	 ah it exits now, must be some type of background job, weird
[21:02:31] <jhathaway>	 *exists
[21:02:51] <vgutierrez>	 ?
[21:03:20] <vgutierrez>	 Each directory is created by acme-chief upon cert issuance
[21:04:24] <vgutierrez>	 Thing is.. the master needs to get the certificate issued and then a systemd timer will mirror that to the passive hosts 
[21:04:49] <vgutierrez>	 So... up to 30 minutes between the CR gets merged and puppet runs on the master host
[21:05:20] <vgutierrez>	 And then up to another 30 minutes till the systemd timer kicks in and rsyncs the new directory to the passive host 
[21:05:51] <jhathaway>	 nod that makes sense, the exception on the passive host is what confused me, and then I looked for a timer, but since I was on a passive host I couldn't find it
[21:06:55] <vgutierrez>	 yeah.. that's right 
[21:07:31] <jhathaway>	 thanks for the help folks