[09:02:35] $log (START) failover sopel-prod boston -> london [09:02:43] Saved item "(START) failover sopel-prod boston -> london" [09:41:10] $log end production failover [09:41:19] Saved item "end production failover" [09:44:32] $log reboot bots3001 for kernel updates [09:44:39] Saved item "reboot bots3001 for kernel updates" [09:48:46] $log upgrade and reboot tools3001 [09:48:56] Saved item "upgrade and reboot tools3001" [09:48:58] $log upgrade and reboot bots2002 [09:49:08] Saved item "upgrade and reboot bots2002" [09:57:51] $log depool and stop mysql on db3001; update both dbs and tools2002 [09:57:58] Saved item "depool and stop mysql on db3001; update both dbs and tools2002" [09:58:15] $log reboot db3001 [09:58:22] Saved item "reboot db3001" [10:04:55] $log reboot db2002 & tools2002 [10:05:01] Saved item "reboot db2002 & tools2002" [10:11:21] $log all services back to intended states [10:11:32] Saved item "all services back to intended states" [11:05:11] !list [11:05:11] botswiki; all; [11:05:23] Why didn't that trigger on my edit [11:08:24] #miraheze-feed spamed maybe ? 🤔 [11:08:54] HeartsDo: yep [11:08:59] It's 90 minutes behind [11:09:02] https://dfgear.miraheze.org/wiki/User_talk:Cedrei [11:09:03] [url] User talk:Cedrei - DragonFable Equipment Repository | dfgear.miraheze.org [11:09:04] [url] User talk:Cedrei - DragonFable Equipment Repository | dfgear.miraheze.org [11:10:03] I think Reception123 will have to restart it [11:11:18] Someone made over 500 edits in less than 10 minutes [12:32:29] [[Upgrades]] on botswiki was edited by RhinosF1 with the following comment: Created page with "Guidance for low downtime updates. # The non-active tools & bots servers should be updated first if possible. These can be rebooted freely. # Decide whether to do a DC failover (only needed for larger Link: https://wiki.fossbots.org/w/index.php?oldid=4907&rcid=1807 [13:16:27] [[Upgrades]] on botswiki was edited by RhinosF1 with the following comment: /* Standard Procedure */ Link: https://wiki.fossbots.org/w/index.php?diff=4908&oldid=4907&rcid=1808 [16:00:58] [[Upgrades]] on botswiki was edited by RhinosF1 with the following comment: /* Traffic Failover */ Link: https://wiki.fossbots.org/w/index.php?diff=4909&oldid=4908&rcid=1809 [16:22:19] [[Upgrades]] on botswiki was edited by RhinosF1 with the following comment: /* DC Failover */ Link: https://wiki.fossbots.org/w/index.php?diff=4910&oldid=4909&rcid=1810 [16:22:33] Sario: ^ [16:23:36] [[Upgrades]] on botswiki was edited by RhinosF1 with the following comment: /* DC Failover */ Link: https://wiki.fossbots.org/w/index.php?diff=4911&oldid=4910&rcid=1811 [16:26:28] Sario: I wouldn't trust the DC failover plan [16:26:36] We'd need a big planning meeting before trying that [16:26:52] Traffic failover v standard failover is pretty much discretion [16:27:12] Depends how concerned you are about phab being down when tools reboots [16:27:20] Or icingaweb [16:27:33] We could ask MacFan4000 how to make icingaweb multi DC [16:31:46] Honestly, I'm not overly concerned. If we can get those services to be redundant without too much fuss, that's great, but otherwise even a day's worth of downtime isn't that big of a deal for us. [16:33:28] Sario: no, I'd say we've done well at low downtime upgrades [16:34:08] DC failovers should be for things like Debian upgrades or emergencies only [16:35:38] The standard is what I did this morning [16:35:43] Id say it works quite well [16:37:14] It would be nice if we didn't have such horrid cross-DC latency [16:37:34] Cloudflare tunnels or something might help [16:37:42] When they get it released