[15:04:47] hold onto your butts [15:05:13] good luck andrewbogott [15:05:59] Going to be a longer outage than I planned because we're setting things to r/o and letting replication settle down first, it was running behind. [15:12:27] 🤞 [15:29:10] !log stopping all VMs on cloudvirt1019, reimaging host [15:29:11] andrewbogott: Unknown project "stopping" [15:29:19] !log admin stopping all VMs on cloudvirt1019, reimaging host [15:29:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:19:43] OK, I believe toolsdb to be back to normal-ish. Would love it if someone else here (e.g. Lucas_WMDE ) can confirm [16:20:35] https://quickcategories.toolforge.org/ looks fine again [16:20:39] though that’s only testing read access [16:20:53] Can you check r/w? It was read only for a while [16:21:01] https://dicare.toolforge.org/lexemes/challenge.php also looks fine again, and that tool apparently writes on GET requests [16:21:10] nice [16:21:13] thank you! [16:24:55] and I just saw a write from QuickCategories too, so it’s definitely working :) [16:25:05] (dicare isn’t mine, it was just mentioned elsewhere) [16:25:14] (as being already broken during the r/o period) [16:29:30] !log tools.quickcategories removed EXPECTED_DATABASE_ERROR from config again now that ToolsDB outage is over [16:29:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [18:06:32] !log wm-bot updated a cloak in the admins config [18:06:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wm-bot/SAL