[09:56:41] <Not-1527>	 [02puppet] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J8lso
[09:56:42] <Not-1527>	 [02puppet] 07RhinosF1 0343d0055 - services: make ssh + ping(4|6) #page
[10:01:00] <icinga-mhbots>	 CUSTOM - ping4 #page on bots1 is UNKNOWN:  rhinosf1 Test
[10:02:00] <RhinosF1>	 If #page is in an alert, treat it as important
[10:02:46] <RhinosF1>	 They'll be directed some place useful soon
[10:03:00] <RhinosF1>	 Probably by Miraheze bot + hund
[10:03:18] <RhinosF1>	 Maybe something like slack
[16:04:49] <Not-1527>	 [02puppet] 07sario528 pushed 031 commit to 03sario528-patch-1 [+0/-0/±1] 13https://git.io/J8gSk
[16:04:51] <Not-1527>	 [02puppet] 07sario528 03f931136 - Add ##Sario
[16:04:52] <Not-1527>	 [02puppet] 07sario528 created branch 03sario528-patch-1 - 13https://git.io/JJGvA
[16:05:27] <Not-1527>	 [02puppet] 07sario528 opened pull request 03#195: Add ##Sario - 13https://git.io/J8gSi
[16:06:08] <Sario>	 RhinosF1: got a moment to review? ^
[16:07:00] <RhinosF1>	 Sario528: missing a ]
[16:07:36] <RhinosF1>	 Sario: *
[16:08:49] <Not-1527>	 [02puppet] 07sario528 pushed 031 commit to 03sario528-patch-1 [+0/-0/±1] 13https://git.io/J8gHd
[16:08:50] <Not-1527>	 [02puppet] 07sario528 03d04e558 - fix
[16:08:51] <Not-1527>	 [02puppet] 07sario528 synchronize pull request 03#195: Add ##Sario - 13https://git.io/J8gSi
[16:10:33] <Sario>	 And Sario528 still fires a notification since it contains the string Sario, it just doesn't get colored
[16:13:23] <Not-1527>	 [02puppet] 07RhinosF1 closed pull request 03#195: Add ##Sario - 13https://git.io/J8gSi
[16:13:25] <Not-1527>	 [02puppet] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J8gdn
[16:13:26] <Not-1527>	 [02puppet] 07sario528 03e0dfa04 - Add ##Sario (#195) * Add ##Sario * fix
[16:13:28] <Not-1527>	 [02puppet] 07RhinosF1 deleted branch 03sario528-patch-1 - 13https://git.io/JJGvA
[16:13:29] <Not-1527>	 [02puppet] 07RhinosF1 deleted branch 03sario528-patch-1
[16:13:40] <Sario>	 RhinosF1: Thanks
[16:13:40] <RhinosF1>	 Sario: ^
[16:13:53] <Sario>	 I'll run puppet and test
[16:13:57] <RhinosF1>	 Don't forget to use the cache reload command
[16:14:24] <Sario>	 yup
[16:21:49] <RhinosF1>	 Sario: missing } at end
[16:21:54] <RhinosF1>	 We should have CI
[16:22:02] <Sario>	 Ahh, I'll fix
[16:24:54] <Not-1527>	 [02puppet] 07sario528 pushed 031 commit to 03sario528-patch-1 [+0/-0/±1] 13https://git.io/J8gju
[16:24:55] <Not-1527>	 [02puppet] 07sario528 03eb5ef74 - fix again
[16:24:57] <Not-1527>	 [02puppet] 07sario528 created branch 03sario528-patch-1 - 13https://git.io/JJGvA
[16:24:58] <Not-1527>	 [02puppet] 07sario528 opened pull request 03#196: fix again - 13https://git.io/J8gj2
[16:27:36] <Not-1527>	 [02puppet] 07RhinosF1 closed pull request 03#196: fix again - 13https://git.io/J8gj2
[16:27:38] <Not-1527>	 [02puppet] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J82vI
[16:27:39] <Not-1527>	 [02puppet] 07sario528 037b834d6 - fix again (#196)
[16:27:41] <Not-1527>	 [02puppet] 07RhinosF1 deleted branch 03sario528-patch-1 - 13https://git.io/JJGvA
[16:27:42] <Not-1527>	 [02puppet] 07RhinosF1 deleted branch 03sario528-patch-1
[16:27:47] <RhinosF1>	 Sario: merges
[16:28:06] <Sario>	 puppet running
[16:42:40] <Sario>	 $log restarting prod
[16:42:47] <Void-bot>	 Saved item "restarting prod"
[16:44:08] <Sario>	 $log restarting beta
[16:44:13] <icinga-mhbots>	 PROBLEM - MirahezeBots sopel on bots1 is WARNING: PROCS WARNING: 1 process with command name 'sopel'   
[16:44:15] <Void-bot>	 Saved item "restarting beta"
[16:46:13] <icinga-mhbots>	 RECOVERY - MirahezeBots sopel on bots1 is OK: PROCS OK: 2 processes with command name 'sopel'   
[16:46:16] <MacFan4000>	 $log restart beta again
[16:46:22] <Void-bot>	 Saved item "restart beta again"
[17:01:52] <RhinosF1>	 We need to plan bullseye
[17:02:08] <RhinosF1>	 It's out in like a week
[17:02:19] <RhinosF1>	 I'll try and reimage test2001 nearly as soon as possible
[17:02:38] <RhinosF1>	 Then start provisioning other servers
[17:02:44] <RhinosF1>	 Mysql failover will be fun
[17:02:57] <RhinosF1>	 And probably need a fair bit of phab downtime
[17:04:05] <RhinosF1>	 MacFan4000: I assume mysqldump & then mysql < is best
[17:04:36] <RhinosF1>	 It might be worth testing latency to the aarch platform
[17:04:54] <RhinosF1>	 Is it possible to set MariaDB up on test2001 and connect them
[17:04:56] <MacFan4000>	 yes, though phabricator has multiple databases
[17:05:17] <RhinosF1>	 MacFan4000: yeah I know it's huge
[17:05:40] <RhinosF1>	 I just want to see how phab will behave when we have db2002 but are still on tools1
[17:06:05] <MacFan4000>	 I have a shell script that can go through and export them all to .sql files - thats how I moved the dbs for testwiki phab
[17:06:43] <MacFan4000>	 and then I have another script that goes through and imports them all
[17:08:15] <RhinosF1>	 MacFan4000: okay, can you test what latency is from tools to test
[17:08:20] <RhinosF1>	 Icinga might know tbh
[17:09:20] <RhinosF1>	 130+
[17:09:36] <RhinosF1>	 So we're gonna see a pretty big performance regression
[17:09:50] <RhinosF1>	 At least tools is easy to failover
[17:09:55] <MacFan4000>	 https://www.irccloud.com/pastebin/UsJfHkyN/
[17:10:02] <MacFan4000>	 there's the output of ping
[17:10:10] <RhinosF1>	 Ye
[17:10:31] <RhinosF1>	 We can probably fail phab over in one go tbh
[17:10:48] <RhinosF1>	 You can operate phab from multiple app servers
[17:11:16] <RhinosF1>	 So it won't be hard to do it in one go
[17:12:43] <RhinosF1>	 If we setup a read only phab-new that connects to db1 using NAT64
[17:13:14] <RhinosF1>	 So image tools2002 first
[17:14:11] <RhinosF1>	 yeah we can have 2 appservers, so 1st pool tools2002
[17:14:35] <RhinosF1>	 Then we can switchover from db1 -> db2002
[17:14:49] <RhinosF1>	 Then we depool tools1
[17:14:59] <RhinosF1>	 That will work
[17:15:34] <Sario>	 We should document this plan
[17:22:14] <RhinosF1>	 Sario: https://phab.mirahezebots.org/T269
[17:22:15] <MirahezeBot>	 [url] ⚓ T269 Failover phab & db | phab.mirahezebots.org