[05:33:50] <marostegui>	 dhinus: https://phabricator.wikimedia.org/T365424#9980124
[07:02:12] <marostegui>	 volans: Let me know when you want to do https://phabricator.wikimedia.org/T369882, there's no time constraint, it can be done anytime today, so whenever works for you, just let me know. No rush
[07:13:13] <volans>	 marostegui: give me just the time to go through the procedure in the task and I'll ping you
[07:21:54] <marostegui>	 volans: no rush, take your time
[07:24:05] <dhinus>	 marostegui: ack, using 1021 as a testbed is a good idea!
[07:24:41] <marostegui>	 dhinus: cool, if you want to coordinate with btullis, that'd be great (I pinged him in the decom task, so he doesn't do it until he's synced with you)
[07:25:00] <volans>	 marostegui: for questions should I just ask here or better in private to not pollute the channel?
[07:25:36] <dhinus>	 marostegui: ok
[07:26:44] <marostegui>	 volans: arnaudb will answer them, as it is a good practice, I'll be reading and if there's something not clear, I can try to help
[07:27:36] <volans>	 sounds good
[07:29:14] <marostegui>	 dhinus: do you have any DB running stretch?
[07:29:19] <marostegui>	 I'd like to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/761927
[07:30:34] <marostegui>	 dhinus: nevermind, it was already done in a different patchset
[07:30:59] <dhinus>	 ok cool :)
[07:31:11] <marostegui>	 dhinus: anything still running buster?
[07:31:42] <volans>	 arnaudb, marostegui: so looking at the preparatory step of checking the config I noticed that the new master has 10.6.16 while the old one 10.6.17 and the latest in the fleet is 10.6.18. Should we consider to upgrade it before switching?
[07:32:50] <dhinus>	 marostegui: that's more likely, let me check 
[07:33:10] <marostegui>	 dhinus: Thanks (it is EOL btw)
[07:33:23] <arnaudb>	 volans: usually we don't bother, but you can try to upgrade the host before to experience a bit more of the server's lifecycle
[07:33:53] <marostegui>	 volans: I'd prefer if we don't, we try not to have a master running a higher version than the replicas (it is not a big deal and will most likely run  fine, but just in case)
[07:34:20] <volans>	 it is already like that :D
[07:34:33] <volans>	 some replicas are .16 and the current master in codfw is .17
[07:34:50] <volans>	 ack, leaving as is
[07:35:01] <marostegui>	 volans: yeah, but most of them are 17. But anyway, you can if you want, it is not a big deal
[07:35:23] <volans>	 nah, let's dilute the excitement, not all together :)
[07:35:29] <marostegui>	 dhinus: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1054271
[07:35:39] <marostegui>	 dhinus: Gave it -2 for now
[07:36:22] <dhinus>	 marostegui: we have a task to upgrade or remove all buster hosts but there are a few left. I think none of those is using the db puppet classes though.
[07:36:39] <dhinus>	 https://os-deprecation.toolforge.org/
[07:36:44] <marostegui>	 dhinus: I can try to merge and if not...we can revert, whatever you prefer
[07:38:24] <dhinus>	 should be safe but let me do a quick search for that puppet class
[07:38:32] <marostegui>	 thanks
[07:41:37] <volans>	 arnaudb, marostegui: ok the procedure looks good. Do we need the final steps too? (From (If needed): Depool db2121 for maintenance.)
[07:42:04] <volans>	 last bit is that one of the final steps says "Change db2121 weight to mimic the previous weight db2218:" but there is no step that says save the current weight :D
[07:42:45] <volans>	 for the rest all looks good and fairly trivial (if there is no issue :D  )
[07:44:58] <arnaudb>	 volans: we rely on phabricator pastes for missing info usually, but it's indeed a missing instruction :D
[07:45:16] <volans>	 should I add it to switchover-tmpl.py ?
[07:45:31] <arnaudb>	 sure!
[07:46:28] <marostegui>	 we don't use switchover-tmpl.py anymore do we?
[07:46:35] <volans>	 doh
[07:46:50] <marostegui>	 volans: you'd need to add it to https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/tree/main/switchmaster?ref_type=heads
[07:46:52] <marostegui>	 those templates
[07:46:56] <volans>	 that's what I found with codesearch, I hoped it was used by the toolforge tool too
[07:47:06] <volans>	 ack
[07:47:06] <volans>	 thx
[07:56:33] <volans>	 template changes for later
[07:56:45] <volans>	 arnaudb, marostegui: I think we can start whenever you want. I've created a tmux on cumin2002, you can attach with: sudo -i tmux attach -rt T369882
[07:56:46] <stashbot>	 T369882: Switchover s7 master (db2121 -> db2218) - https://phabricator.wikimedia.org/T369882
[07:57:39] * arnaudb is attached
[07:57:58] <volans>	 the only request I have is that because I'm also oncall, if anything else happen I might ask you to takeover
[07:58:35] <arnaudb>	 ack, be sure to mention the step/state at which you let things, no problem otherwise!
[08:00:03] <marostegui>	 volans: I will be ready in like 10 minutes, but arnaudb has done this a million times, so there's no need to wait for me anyway
[08:00:59] <volans>	 ok
[08:02:20] <volans>	 from the config diff, anything special I should check? the semi-sync bits are expected and there is no statement/row line so they have both the same (and I've checked on orchestrator they have both statement
[08:02:59] <arnaudb>	 I generally try to spot any major difference (i.e. version, semi_sync, missing field)
[08:03:19] <volans>	 do you see anything unexpected here?
[08:03:47] <arnaudb>	 nothing raising my attention outside of what you said about minors
[08:04:14] <volans>	 ok to proceed then?
[08:04:19] <arnaudb>	 yes!
[08:04:49] <volans>	 why do we downtime also eqiad's hosts?
[08:05:19] <arnaudb>	 I think this is a safety measure to avoid alarms while running the procedure, given that we're supposed to be monitoring the situation 
[08:05:30] <volans>	 k
[08:05:30] <arnaudb>	 but I'd be glad to have a refresher from marostegui on this ↑
[08:06:15] <marostegui>	 volans: I checked earlier today and it was all as expected 
[08:06:21] <marostegui>	 so you're good 
[08:06:25] <volans>	 <3
[08:08:38] <volans>	 arnaudb: let me know how often do you want me to stop before proceeding ;)
[08:08:53] <arnaudb>	 I keep an eye on your session volans 
[08:08:59] <arnaudb>	 if you don't see me scream, that's ok
[08:09:44] <volans>	 lol, ok, but I wouldn't know if you stop looking :D
[08:10:24] <arnaudb>	 I keep the terminal on display so I don't! if it moves/stops moving I check
[08:11:31] <volans>	 edit looks good?
[08:11:36] <arnaudb>	 yup
[08:11:39] <volans>	 I see that the candidate master bit is changed later
[08:12:02] <arnaudb>	 it is still a candidate master!
[08:12:20] <volans>	 correct
[08:12:21] <arnaudb>	 the coup is yet to happen 
[08:13:43] <volans>	 how safe is this magic box? :D
[08:13:51] <arnaudb>	 a lot to very
[08:14:06] <volans>	 I just monitor on orchestrator that it does what it promises?
[08:14:25] <arnaudb>	 indeed, I also tend to keep a tab open with alertmanager for peace of mind
[08:14:45] <volans>	 any specific filter on AM?
[08:15:11] <arnaudb>	 I usually discard warnings and try to narrow down to DP 
[08:15:18] <arnaudb>	 nothing fancy
[08:16:12] <volans>	 ok
[08:17:08] <volans>	 proceeding with the replicas move
[08:17:21] <volans>	 great start :D
[08:17:30] <arnaudb>	 it failed on the select @@read_only;
[08:17:42] <arnaudb>	 first time I see this 🤔
[08:18:41] <volans>	 they are noth RO because we're in codfw, so I guess all expected
[08:18:49] <marostegui>	 yep
[08:18:59] <marostegui>	 that's why we pass --read-only-master
[08:19:08] <marostegui>	 btw $  sudo -i tmux attach -rt T369882
[08:19:08] <marostegui>	 no sessions
[08:19:08] <stashbot>	 T369882: Switchover s7 master (db2121 -> db2218) - https://phabricator.wikimedia.org/T369882
[08:19:17] <arnaudb>	 on cumin2002 marostegui 
[08:19:20] <arnaudb>	 not 1002
[08:19:23] <marostegui>	 ah! thanks!
[08:19:24] <arnaudb>	 (i did the same :p)
[08:19:26] <volans>	 does it have some debug logging saved somewhere? I don't see any debug option
[08:19:51] <arnaudb>	 nah it's quite rigid afaik
[08:19:58] <arnaudb>	 I haven't had to debug with it yet
[08:21:26] <marostegui>	 you guys want to try yourselves or you want me to help with the troubleshooting?
[08:21:27] <volans>	 it failed here https://gitlab.wikimedia.org/repos/sre/wmfmariadbpy/-/blob/main/wmfmariadbpy/cli_admin/switchover.py#L107
[08:21:55] <volans>	 I've executed manually the query on both and it's ok
[08:22:55] <volans>	 marostegui: is it possible it's just credential issue for the way I'm running things?
[08:23:14] <marostegui>	 volans: I just checked and I can reach them fine
[08:23:18] <volans>	 (sudo -i)
[08:24:28] <volans>	 I would retry the command just to ensure is a hard failure and not transient
[08:24:28] <marostegui>	 volans: can you try from cumin1002?
[08:24:30] <marostegui>	 so we can isolate?
[08:24:33] <volans>	 sure
[08:24:35] <marostegui>	 volans: that also works yeah
[08:24:49] <marostegui>	 https://phabricator.wikimedia.org/P66476 for what is worth
[08:24:53] <volans>	 ok same error again on cumin2002 trying on 1001
[08:24:55] <volans>	 *1002
[08:25:10] <volans>	 yep I've checked that too
[08:26:27] <volans>	 created same tmux on 1002 too
[08:26:43] * arnaudb (attached)
[08:26:49] <volans>	 ready to run once you're attached
[08:27:05] <arnaudb>	 go!
[08:27:21] <volans>	 and here it works
[08:27:27] <marostegui>	 that's interesting
[08:27:35] <marostegui>	 we need to check what it is
[08:27:48] <volans>	 indeed
[08:27:53] <marostegui>	 it is worrying
[08:28:07] <marostegui>	 let's finish the switch and then we can double check
[08:28:11] <volans>	 sure
[08:29:09] <volans>	 it has sterted to move hosts
[08:31:40] <volans>	 why the lag of the replicas doesn't goes down after the move?
[08:31:52] <marostegui>	 dhinus: I've merged, if you see something strange let me know
[08:32:34] <marostegui>	 volans: because the replication on the master is still stopped (as expected)
[08:32:40] <marostegui>	 once it is started, it will go down
[08:33:21] <dhinus>	 marostegui: my "quick search" became a bit longer as I got lost trying to use Cumin :D volans: is there a way to do something like P:mariadb::packages_wmf with cloudvps hosts?
[08:33:26] <volans>	 sure, I meant when started, I see it stops and starts for each hosts, but apparently the others catched up, just db2220 is a bit slower to catch up
[08:33:47] <arnaudb>	 sometimes transactions take a bit more time to be handled
[08:34:03] <arnaudb>	 there is no real predictability on replag recovery
[08:34:09] <volans>	 dhinus: https://openstack-browser.toolforge.org/puppetclass/profile::mariadb::packages_wmf :-P ???
[08:34:34] <volans>	 also T179816
[08:34:34] <stashbot>	 T179816: Cumin: create external backend for WMCS Puppet API - https://phabricator.wikimedia.org/T179816
[08:37:03] <dhinus>	 volans: I'm not sure that page is actually accurate, it only includes direct references, not indirect ones through other classes
[08:37:12] <volans>	 I know :D
[08:37:40] * dhinus subscribes to T179816
[08:37:43] <volans>	 for projects with their own puppetmaster and pupeptdb you can run cumin within the project and use the same query (like deployment-prep)
[08:37:53] <volans>	 dhinus you have to implement it, not subscribe :D
[08:38:05] <volans>	 it's not in the TODO there was no interest by WMCS on that at the time
[08:38:23] <volans>	 and if it doesn't report properly the things it might be misleading
[08:38:23] * dhinus likes, subscribes and hits the bell button
[08:38:47] <volans>	 replicas move status: 3 to go
[08:38:54] <Emperor>	 sounds like volunteering to me, dhinus ;p
[08:39:19] <volans>	 +1
[08:41:42] <volans>	 last one to go
[08:42:12] <marostegui>	 Emperor: hahaha
[08:42:38] <dhinus>	 volunteering for many things equates to volunteering for none :P
[08:43:53] <arnaudb>	 looks good volans
[08:44:15] <volans>	 the semi-sync warning is expected?
[08:44:26] <arnaudb>	 yeah it's not enabled everywhere
[08:44:34] <volans>	 orchestrator looks good to me
[08:44:57] <volans>	 nothing screaming in AM 
[08:45:16] <arnaudb>	 yep, everything looks ok!
[08:46:03] <volans>	 ok continuing with teh safe stuff
[08:46:45] <volans>	 tbhe patch is https://gerrit.wikimedia.org/r/c/operations/puppet/+/1053827, review welcome :)
[08:47:15] <arnaudb>	 +1ed!
[08:48:22] <volans>	 ack merging
[08:48:57] <arnaudb>	 next steps are faster than the first ones
[08:49:08] <volans>	 ack
[08:49:49] <volans>	 puppet merge completed
[08:50:38] <volans>	 are we a GO for the actual switch? :)
[08:50:51] <arnaudb>	 lgtm volans ! jump when you feel like it :)
[08:51:27] <volans>	 ack going
[08:51:54] <marostegui>	 🤞🤌
[08:52:12] <volans>	 :D
[08:52:22] <volans>	 seems all good, saying yes
[08:52:37] <volans>	 arnaudb: looks good to you?
[08:52:46] <arnaudb>	 yep!
[08:53:38] <arnaudb>	 famous curl printed here
[08:53:52] <volans>	 yes but for later
[08:54:01] <arnaudb>	 (it's in the doc now, you can let it slide)
[08:54:31] <arnaudb>	 lgtm
[08:54:36] <arnaudb>	 monitoring confirms
[08:54:58] <volans>	 agree
[08:55:19] <volans>	 orchestrator too, jsut a bit of lag
[08:55:40] <volans>	 although show slave status showed 0, but I guess they measure it in a different way
[08:55:54] <volans>	 ok to proceed with dbctl?
[08:56:23] <arnaudb>	 yep
[08:56:45] <volans>	 LGTM
[08:56:50] <arnaudb>	 same!
[08:57:15] <volans>	 running puppet
[08:57:50] <arnaudb>	 after the curl, replag seen on orchestrator will be trustworthy again
[08:57:56] <volans>	 ok
[08:58:08] <arnaudb>	 atm it's all aske
[08:58:09] <arnaudb>	 w
[08:58:45] <volans>	 cleanup tasks
[08:59:48] <arnaudb>	 orch back in order
[08:59:57] <volans>	 nice
[09:00:17] <volans>	 isn't missing a dbctl commit?
[09:00:32] <arnaudb>	 sudo dbctl config commit -m "Depool db2121 T369882" at the end
[09:00:33] <stashbot>	 T369882: Switchover s7 master (db2121 -> db2218) - https://phabricator.wikimedia.org/T369882
[09:00:34] <arnaudb>	 but indeed
[09:00:42] <volans>	 ok waiting
[09:00:48] <arnaudb>	 nah go for it
[09:00:55] <arnaudb>	 it's a missing instruction
[09:01:11] <arnaudb>	 oh no, wait
[09:01:14] <arnaudb>	 it's a noop change
[09:01:18] <arnaudb>	 so you don't have to commit
[09:01:25] <volans>	 right it's just metadata
[09:01:32] <volans>	 noop for mw config
[09:01:35] <arnaudb>	 yep
[09:02:00] <volans>	 zarcillo seems correct
[09:02:05] <arnaudb>	 agreed
[09:02:17] <volans>	 do we need mainteanance on db2121?
[09:02:38] <arnaudb>	 don't bother, schema change will depool it automatically
[09:03:14] <volans>	 but if I set the weight without depooling it will be pooled directluy with full weight right?
[09:03:33] <volans>	 and it has only warm cache as a master, not as a replica (I guess query patterns are quite different)
[09:03:54] <arnaudb>	 the weight at which it's pooled will help warming up the cache
[09:04:22] <arnaudb>	 but it's depooled atm
[09:04:31] <marostegui>	 volans: yes, thr order is correct
[09:04:34] <marostegui>	 We should first depool
[09:04:38] <marostegui>	 Then change the weight 
[09:04:46] <marostegui>	 And then repool (ideally in small steps)
[09:04:52] <volans>	 ok so the (If needed): should be removed?
[09:04:59] <marostegui>	 Otherwise you are going from 0 to who knows what 
[09:05:04] <volans>	 exactly
[09:05:48] <marostegui>	 volans: so normally the switchover is done for a schema change, so you can just depool, change the weight and ping whoever needed the schema change (me)
[09:05:56] <marostegui>	 So I'll run the schema change and repool when done 
[09:06:06] <volans>	 ack
[09:06:20] <volans>	 I'm restoring the old weight including API ok?
[09:06:26] <arnaudb>	 yep
[09:07:50] <volans>	 arnaudb: LMW if edit looks ok
[09:08:07] <arnaudb>	 lgtm!
[09:09:43] <volans>	 edit does the commit too, weird, there was no commit message
[09:10:16] <arnaudb>	 it's a noop change iirc
[09:10:20] <arnaudb>	 since the host is depooled
[09:10:24] <arnaudb>	 so no commit here
[09:11:26] <volans>	 the diff is there https://phabricator.wikimedia.org/P66478
[09:11:29] <volans>	 and it did commit
[09:11:41] <volans>	 I just didn't recall that dbctl edit does commit automatically
[09:11:47] <volans>	 no two step as usual
[09:12:01] <volans>	 sorry I'm stupid
[09:12:06] <volans>	 mixing lines in the output
[09:12:14] <marostegui>	 volans: we forgive you for that
[09:12:31] <volans>	 ofc it's a noop
[09:13:04] <volans>	 marostegui: all done, all yours, task updated until the schema change
[09:13:21] <marostegui>	 volans: you can just remove that step andclose the task as fixed
[09:13:29] <marostegui>	 thank you for doing it :)
[09:14:34] <volans>	 thank you, it was helpful and we found an issue from cumin2002 :D
[09:14:45] <marostegui>	 yeah, volans arnaudb can you try to see what the issue there could be?
[09:15:26] <volans>	 sure
[09:16:01] <marostegui>	 thanks :*
[10:03:05] <volans>	 if I read correctly the code, db-swithover does ask for confirmation only after the slave move, so a "safe" command to test it could be this one (that I guess will fail anyway in the preflight because the new master is not a replica of the old one)
[10:03:11] <volans>	 db-switchover --replicating-master --read-only-master --skip-slave-move db2121 db2218
[10:03:19] <volans>	 (same command of the switch we did today)
[10:03:38] * volans will also put a print and exit, but to be on teh safe side
[10:06:41] <volans>	 [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain
[10:07:31] <volans>	 {hotfix removed, will send patch}
[10:08:00] <marostegui>	 volans: should we capture this on a task so there's a record of it?
[10:10:47] <volans>	 yeah
[10:10:52] <volans>	 I'll open one]
[10:10:53] <arnaudb>	 creating the ticket
[10:11:01] <arnaudb>	 ah
[10:11:08] <volans>	 go ahead
[10:11:11] <arnaudb>	 ack!
[10:13:06] <arnaudb>	 https://phabricator.wikimedia.org/T370029
[10:15:09] <volans>	 thx updated
[11:01:17] <kwakuofori>	 hello everyone, back in the office. let me know here on in private if I need to do anything for you!
[11:01:28] <volans>	 welcome back!
[11:01:36] <marostegui>	 kwakuofori: welcome back
[11:01:45] <kwakuofori>	 thanks, volans! welcome to you!
[11:01:45] <Emperor>	 hi. Good break?
[11:01:56] <kwakuofori>	 marostegui: thanks!
[11:02:09] <kwakuofori>	 Emperor: it was indeed! thanks for asking
[11:02:22] <Emperor>	 👍
[11:33:42] <volans>	 my findings so far: https://phabricator.wikimedia.org/T370029#9980901
[12:37:46] <arnaudb>	 volans: it looks like it relates to this issue: https://phabricator.wikimedia.org/T355157
[12:39:01] <volans>	 arnaudb: in which way?
[12:40:19] <arnaudb>	 we had issue with the certificate bundle on orchestrator, which was the origin of verify=False flag 
[12:43:11] <volans>	 I've tried with True and works fine for me
[12:45:00] <arnaudb>	 ack
[14:50:17] <arnaudb>	 fyi I've restarted s6's dump, it's been failed because of T367781
[14:50:17] <stashbot>	 T367781: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781