[08:19:12] <marostegui>	 Hello, I just saw the email about holding MW deployments, but I would need to deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/817195 for https://phabricator.wikimedia.org/T313382
[08:19:17] <marostegui>	 So I am wondering if I can proceed or not
[08:21:02] <volans>	 marostegui: the current problem is with scap that doesn't restart php-fpm on the canaries
[08:21:12] <volans>	 so I think you're totally fine
[08:21:44] <marostegui>	 Roger, thank you volans 
[08:22:00] <marostegui>	 Also you are oncall, so I can proceed blindly 
[08:22:09] <volans>	 lol
[08:22:51] <vgutierrez>	 in volans we trust!
[08:22:58] <vgutierrez>	 (blindly)
[08:42:28] <_joe_>	 marostegui: I'm working on it
[08:42:42] <marostegui>	 _joe_: <3
[08:42:51] <_joe_>	 volans: we *need* to restart php manually on the canaries if we do deployments now
[08:46:00] <volans>	 _joe_: yes, I'm aware, manuel's patch is not touching anything scap related, hence my good to go :)
[08:46:24] <marostegui>	 volans: But I do deploy with scap sync-file
[08:46:43] <volans>	 marostegui: the above patch is in the puppet repo... wrong link?
[08:46:52] <marostegui>	 volans: oh yes :(
[08:46:55] <volans>	 ahhhh
[08:47:00] <marostegui>	 https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/817194
[08:47:03] <marostegui>	 sorry :(
[08:47:05] <marostegui>	 I deployed that already
[08:47:37] <volans>	 ok, then we need to restart the php-fpm on the canaries manually, but I'm sure joe has the cumin command handy :D
[08:50:37] <volans>	 marostegui: I *think* you can use restart-php-fpm-all as command and A:mw-canary as cumin alias
[08:50:44] <volans>	 _joe_: is the above correct? ^^^
[08:51:07] * marostegui waits
[08:53:26] <volans>	 I'm not sure if you need to batch them or not
[09:10:12] <_joe_>	 volans: yes
[09:10:27] <_joe_>	 volans: batching you can do or... you cna trust poolcounter
[09:10:48] <_joe_>	 but if you wait ~ 15 minutes, jnuche and I got to the bottom of what is not working right now
[09:10:57] <volans>	 yeah that's why I wasn't sure, just the canaries and they have poolcounter anyway :)
[09:10:59] <_joe_>	 and I have a puppet fix
[09:11:31] <volans>	 good, no prob for me. Just be aware that marostegui already merged the patch (because of the confusion with the wrong link above)
[09:11:37] <_joe_>	 ok
[09:11:49] <_joe_>	 marostegui: did you restart php on the canaries?
[09:11:53] <_joe_>	 i assume not
[09:12:03] <volans>	 I think was waiting for your confirmation on the above commands
[09:12:14] <_joe_>	 volans: go on please :)
[09:12:20] <volans>	 ok, doing
[09:13:27] <volans>	 {done}
[09:16:49] <marostegui>	 _joe_: Nope, I was waiting indeed
[09:16:55] <marostegui>	 Thank you both
[09:21:04] <volans>	 anytime :)
[09:27:47] <Emperor>	 If I want to mention two bugs in a CR, should I use >1 Bug: pseudo-header, or Bug:Txxx,TYYY ?
[09:28:15] <volans>	 multiple Bug: lines
[09:28:23] <marostegui>	 Emperor: I normally put Bug: TTT and below Bug: TXXXX etc
[09:28:44] <Emperor>	 thanks :) I couldn't manage the requisite google-fu to find the answer 
[09:54:12] <klausman>	 jbond: \o I think ssh_known_hosts generation is currently broken on HEAD. At least for _some_ hosts
[09:54:23] <klausman>	 And seeing as you made changes there yesterday... :)
[09:54:51] <klausman>	 https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/36380/console is a run as of HEAD a few minutes ago, against ores1001
[09:56:57] <elukey>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/816772 was merged yesterday, possibly related
[09:57:00] <elukey>	 jbond --^
[09:57:33] <jbond>	 sorry on a meeting now but wull take a look in a sec
[09:57:38] <klausman>	 TIA
[09:58:11] <klausman>	 elukey: that's a revert, but yes, the last commit on the file mentioned in the error. Though I wonder how a revert would break stuff.
[10:00:10] <klausman>	 Maybe https://gerrit.wikimedia.org/r/c/operations/puppet/+/816724 needs reverting as well. I figure it's possible that the (now deleted) `select` could handle Nil as an input, and created something `join` was then fine with
[10:00:23] <elukey>	 klausman: the parameters variable is probably missing when evaluating the erb file, https://gerrit.wikimedia.org/r/c/operations/puppet/+/816772/2/modules/profile/manifests/ssh/client.pp
[10:01:24] <klausman>	 You're right. That `ensure` logic is probably the linchpin
[10:03:33] <elukey>	 not super urgent, let's wait for John's feedback
[10:03:35] <elukey>	 :)
[10:03:43] <jbond>	 sorry here looking now i think i allready have a fix for this
[10:04:17] <jbond>	 klausman: just to confirm i thik this is only affecting pcc right?
[10:04:24] <jbond>	 ir did you see the issue on a server as well
[10:05:14] <klausman>	 Correct, I've only seen it on pcc
[10:05:19] <jbond>	 ack thanks
[10:19:08] <jbond>	 elukey: klausman: jelto: i think pcc should be fixed now (https://gerrit.wikimedia.org/r/c/operations/puppet/+/816850) let me know if you still see issues
[10:19:33] <klausman>	 thx! My pcc checks a minute or two ago went as expected (shwoing _my_ errrors :))
[10:22:13] <jbond>	 great :)
[10:22:56] <jelto>	 jbond: pcc looks fine now, thanks a lot :)
[10:23:11] <jbond>	 cool :)
[12:24:41] <elukey>	 jbond: thanks!
[13:57:57] <vgutierrez>	 Rook: happy WMFversary ;P 
[13:58:12] <Rook>	 :) thank you!
[14:45:00] <marostegui>	 Krinkle: This is going to be an interesting one: https://phabricator.wikimedia.org/T313811
[14:48:45] <_joe_>	 marostegui: it's hosting stuff that was in redis before?
[14:49:15] <marostegui>	 _joe_: it is hosting all the mainstash stuff
[14:49:21] <_joe_>	 then I can answer your question myself - it can go read-only, we did so during the dc switchover
[14:49:27] <marostegui>	 I guess we can put codfw in RO 
[14:49:29] <marostegui>	 And not eqiad
[14:49:40] <_joe_>	 you can also evict data randomly, no one will notice
[14:49:45] <marostegui>	 I just tested dbctl and it is accepted as an option
[14:49:49] <marostegui>	 so I guess we should be fine
[14:49:49] <_joe_>	 or lose a full shard
[14:50:04] <_joe_>	 so 1/18th of the data
[14:50:06] <_joe_>	 it's also ok
[14:50:28] <_joe_>	 or if your replica to codfw is completely broken and replicates random statements but not all
[14:50:30] <_joe_>	 also ok
[16:26:53] <andrewbogott>	 moritzm: I think we've set up everything we need to for dhinus but the sso dialog (e.g. for icinga.wikimedia.org) is still turning him away. Can you help us understand what piece is missing?
[16:27:02] <andrewbogott>	 username FNegri
[16:34:15] <cdanis>	 andrewbogott: try https://idp.wikimedia.org/logout and then logging back in?
[16:34:57] <dhinus>	 cdanis: that worked, thanks!
[16:35:07] <andrewbogott>	 \o/
[16:35:17] <cdanis>	 I think group memberships are only fetched when you auth, or something
[16:37:48] <mutante>	 optionally let him try to run a command in Icinga. often they will be able to login but are missing those privs
[16:38:48] <andrewbogott>	 hey mutante I was just about to ping you :)  There's a thing on the onboarding list about adding an email address to exim, do you know is that in the mediawiki.org file?  Or something else?
[16:39:02] <andrewbogott>	 + also what is that for?
[16:39:29] <mutante>	 andrewbogott: do you have the full text / what does it say?
[16:39:46] <mutante>	 probably means to add yourself to root@ and other aliases
[16:40:08] <dhinus>	 " add to root@ alias in exim (private.git make sure to use your email username, not shell) can access cloudcontrol1003.wikimedia.org"
[16:40:25] <mutante>	 ah, yea. so this is in   puppetmaster1001:/srv/private/modules/privateexim/files/wikimedia.org
[16:40:28] <mutante>	 in the private repo
[16:40:37] <mutante>	 can be used to test committing in private repo
[16:40:44] <andrewbogott>	 great, that's where we were :)
[16:41:01] <mutante>	 you can add yourself to root@ or optionally other things
[16:41:07] <mutante>	 dns-admin@ peering@ or whatnot
[16:41:38] <andrewbogott>	 got it, thanks mutante !
[16:41:48] <mutante>	 if you want you can ssh to mx1001.wikimedia.org and run puppet there after a commit
[16:41:55] <mutante>	 to see the actual exim change
[16:42:05] <andrewbogott>	 cool
[16:55:16] <dhinus>	 thanks, exim change done, I also ran puppet on mx1001
[16:55:47] <dhinus>	 while I'm here, could anyone please approve my subscription requests for ops@lists.wikimedia.org and ops-private@ ?
[17:17:25] <legoktm>	 dhinus: the "Administrator" account password is in pwstore if you want to approve it yourself :)
[17:28:18] <dhinus>	 legoktm: thanks, will do that tomorrow!
[17:34:38] <mutante>	 do the -owner special addresses still exist? you can ping the list admins at  ops-owner@lists and ops-private-owner@lists if they do
[17:36:24] <legoktm>	 yes, they do
[17:40:31] <mutante>	 dhinus: those addresses specifically go to the specific list owners
[17:47:21] <dhinus>	 thanks mutante! I'll send an email then :)