[06:41:12] <marostegui>	 I am going to switchover codfw s1 master (https://phabricator.wikimedia.org/T290865#7439279)
[07:33:06] <Emperor>	 Job ad is up - https://boards.greenhouse.io/wikimedia/jobs/3565140?gh_src=f5694baa1us
[08:09:39] <Emperor>	 depooling swift{,ro} from codfw so I can add load per T288458
[08:09:40] <stashbot>	 T288458: Put ms-be20[62-65] in service - https://phabricator.wikimedia.org/T288458
[08:19:36] <kormat>	 marostegui: is it safe to run a schema change against s8/codfw now?
[08:19:41] <kormat>	 i see you're breaking s1 today instead
[08:26:31] <Emperor>	 Hm, putting commit ids in commit messages doesn't seem to do anything useful (gerrit turns them into a link that just refers back to the CR in progress); and if I edit the commit to include the Change-Id instead that results in a link that says "no changes". Oh well. 
[08:26:31] <marostegui>	 yes!
[08:26:35] <marostegui>	 safe to run
[08:28:00] <volans>	 Emperor: what are you trying to do?
[08:33:26] <Emperor>	 volans: see the change message in https://gerrit.wikimedia.org/r/c/operations/software/swift-ring/+/731878 - I wanted to refer the reviewer back to commit  a31b690a19a72e8e8dffd243ed4ebceddcdd7390 where ssd_weight was set to 311; gerrit has made a hyperlink of that commit hash, but it just links back to this CR
[08:36:40] <volans>	 interstingly linking the change ID (that usually is the way to go) of that commit says no change
[08:36:59] <Emperor>	 yeah, I tried that and it was even more confusing :-/
[08:37:05] <volans>	 do you/go.dog ever force-push to this repo?
[08:37:26] <Emperor>	 I haven't, dunno about go.dog 
[08:38:05] <volans>	 because the Change-ID should always link to the related CR so this is weird 
[08:38:31] <volans>	 like if it was pushed (even without force) without passing through gerrit
[08:38:49] <godog>	 I might have force-pushed in the past yes, can't remember atm
[08:38:56] <volans>	 in that case you can't link it ;)
[08:39:14] <godog>	 sth else I had done before realizing it wasn't the right thing is reusing commit messages, which included change-id of course
[08:39:17] <Emperor>	 right, so the answer is I should be using Change-ID, that just didn't work on this occasion
[08:39:24] <volans>	 Emperor: you could link
[08:39:25] <volans>	 https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/swift-ring/+/a31b690a19a72e8e8dffd243ed4ebceddcdd7390%5E%21/
[08:39:28] <volans>	 if you want in this case
[08:39:43] <volans>	 or just the commit 
[08:39:44] <volans>	 https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/swift-ring/+/a31b690a19a72e8e8dffd243ed4ebceddcdd7390
[08:40:48] <volans>	 Emperor: correct :)
[08:41:11] <Emperor>	 ta, noted for future reference
[08:41:43] <Emperor>	 now I can be confused by regular things like why account.builder has 100.0 weights in eqiad and 92.0 weights in codfw...
[08:43:10] <godog>	 I'll reply on the review but tl;dr is histerical raisins
[08:44:29] * Emperor goes to update the CR to have nonobject_weight back to 92
[08:54:25] <Emperor>	 (a bit fiddlier than expected)
[08:55:24] <kormat>	 🎻
[09:00:02] <Emperor>	 :p
[09:14:19] <marostegui>	 I have switched codfw s1 master
[09:14:23] <marostegui>	 I am doing clean up tasks now
[09:14:29] <marostegui>	 and making sure everything is fine
[09:17:52] <marostegui>	 kormat: you can proceed with the schema changes on codfw s1, the new master is db2103
[09:18:06] * kormat senses a trap
[09:23:40] <godog>	 "you are in a maze of twisty little passages, all alike"
[09:30:01] <marostegui>	 kormat: are you running a schema change on s1 now?
[09:30:15] <kormat>	 marostegui: it's just finished running, replicating now.
[09:30:21] <marostegui>	 ah cool
[09:30:25] <marostegui>	 I was scared as I saw lots of lag
[09:30:54] <kormat>	 good. :)
[09:34:05] <marostegui>	 I was finally able to close https://phabricator.wikimedia.org/T273280
[09:34:23] <kormat>	 wooo! :D
[09:34:44] <kormat>	 only 10 months to roll out a kernel upgrade. new record!
[10:21:51] <Emperor>	 ...
[11:50:37] <marostegui>	 Emperor: it is nice not to have to restart prometheus exporter anymore, thanks for the work there
[13:03:02] <Emperor>	 you're welcome :)
[14:06:34] <kormat>	 marostegui: trying to appease our demanding users ;) https://gerrit.wikimedia.org/r/c/operations/puppet/+/731977/
[14:07:43] <marostegui>	 I will check in a bit
[15:29:08] <kormat>	 marostegui: any issues with me running some more schema changes against codfw?
[15:29:54] <marostegui>	 kormat: you can go ahead
[15:30:02] <kormat>	 damn
[21:07:01] <ottomata>	 hiyo, for https://phabricator.wikimedia.org/T284150, I'm looking for a recent best practices example in puppet of running a multi instance mysql server.  Am poking around, but is there one you'd recommend I try to emulate?