[00:06:43] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 57543MiB (12% inode=99%); [00:07:50] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 57080MiB (12% inode=99%); [00:07:55] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57779MiB (12% inode=99%); [01:23:31] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=lhmnwiki --reindexAndRemoveOk --indexIdentifier now (END - exit=0) [01:23:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:23:54] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=lhmnwiki (END - exit=0) [01:23:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:12] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=cgwiki --reindexAndRemoveOk --indexIdentifier now (END - exit=256) [01:24:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:57] !log curl -X DELETE https://opensearch-mw.wikitide.net/cgwiki_content https://opensearch-mw.wikitide.net/cgwiki_general [01:25:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:25:16] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=cgwiki --startOver (END - exit=0) [01:25:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:25:33] !log [blankeclair@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=cgwiki (END - exit=0) [01:25:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:42:14] !log UPDATE mw_permissions SET perm_permissions = JSON_REMOVE(perm_permissions, JSON_UNQUOTE(JSON_SEARCH(perm_permissions, 'one', 'checkuser-temporary-account-no-preference'))) WHERE JSON_SEARCH(perm_permissions, 'one', 'checkuser-temporary-account-no-preference') IS NOT NULL; [04:42:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:42:19] !log UPDATE mw_permissions SET perm_permissions = JSON_REMOVE(perm_permissions, JSON_UNQUOTE(JSON_SEARCH(perm_permissions, 'one', 'checkuser-temporary-account-log'))) WHERE JSON_SEARCH(perm_permissions, 'one', 'checkuser-temporary-account-log') IS NOT NULL; [04:42:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:42:24] !log UPDATE mw_permissions SET perm_permissions = JSON_REMOVE(perm_permissions, JSON_UNQUOTE(JSON_SEARCH(perm_permissions, 'one', 'checkuser-temporary-account-auto-reveal'))) WHERE JSON_SEARCH(perm_permissions, 'one', 'checkuser-temporary-account-auto-reveal') IS NOT NULL; [04:42:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:42:30] !log UPDATE mw_permissions SET perm_permissions = JSON_REMOVE(perm_permissions, JSON_UNQUOTE(JSON_SEARCH(perm_permissions, 'one', 'checkuser-temporary-account'))) WHERE JSON_SEARCH(perm_permissions, 'one', 'checkuser-temporary-account') IS NOT NULL; [04:42:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:53:59] PROBLEM - mw153 Puppet on mw153 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [04:54:09] PROBLEM - mw191 Puppet on mw191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [04:57:55] PROBLEM - mw192 Puppet on mw192 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [04:58:26] when are we gonna have enbywiki [05:01:13] PROBLEM - mw163 Puppet on mw163 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [05:01:33] PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [05:03:47] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [05:05:58] PROBLEM - mw201 Puppet on mw201 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [05:06:50] miraheze/python-functions - dependabot[bot] the build passed. [05:07:43] PROBLEM - mw183 Puppet on mw183 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [05:07:59] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [05:08:38] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [05:11:53] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [05:13:09] ^ thats me just ignorr [05:13:10] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [05:13:11] PROBLEM - mw202 Puppet on mw202 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [05:13:14] ignore [05:13:30] PROBLEM - mw203 Puppet on mw203 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [05:14:09] minor amount of CA :3 [05:14:25] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [05:15:03] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [05:15:51] PROBLEM - mwtask161 Puppet on mwtask161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [05:16:15] PROBLEM - mw173 Puppet on mw173 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [05:16:28] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 24 seconds ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [05:16:29] !log [universalomega@mwtask181] starting deploy of {'world': True, 'versions': '1.44'} to all [05:16:30] PROBLEM - mwtask151 Puppet on mwtask151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [05:16:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:18:42] should probably axe femiwiki anyway [05:21:07] PROBLEM - mw193 Puppet on mw193 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.44/skins/Femiwiki/node_modules] [05:21:27] pretty sure its disabled because our install is broken not the skin lol [05:21:47] it is a maintained skin [05:21:58] RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:22:08] RECOVERY - mw191 Puppet on mw191 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [05:22:10] oh that's fun then [05:22:25] !log [universalomega@mwtask181] finished deploy of {'world': True, 'versions': '1.44'} to all - SUCCESS in 355s [05:22:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:24:27] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [05:25:33] !log [universalomega@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44'} to all [05:25:35] !log [universalomega@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [05:25:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:25:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:26:05] @rhinosf1 im going to disable GrowthExperiments on 1.44 [05:26:10] RECOVERY - mw192 Puppet on mw192 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [05:26:13] in LS [05:26:33] Ok [05:26:39] I thought it already was tbh [05:27:54] oh it is [05:28:27] but that doesnt fix it so l10n can run... [05:28:50] it has to be completely uninstalled or CommunityConfiguration installed [05:29:05] RECOVERY - mw163 Puppet on mw163 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [05:29:30] @rhinosf1 what way do you think we should go? [05:29:41] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:30:07] Eventually it's getting uninstalled. We can't vary mediawiki-repos yet so it just delete it from disk each time. [05:30:13] yes we can [05:30:46] @rhinosf1 mediawiki-repos does support only installing extensions on specific versions just not what branch per version. [05:31:06] Then do that [05:31:48] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [05:32:45] Done: https://github.com/miraheze/mediawiki-repos/commit/fe2b659090f909ebc0298eb9dce9242f26673408 [05:32:45] [GitHub] [miraheze/mediawiki-repos] Universal-Omega: Only install GrowthExperiments on 1.43 | 1 change in 1 file | Authored 22 seconds ago | Committed 22 seconds ago [05:33:30] !log [universalomega@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44'} to all [05:33:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:57] RECOVERY - mw201 Puppet on mw201 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [05:34:11] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [05:35:36] RECOVERY - mw183 Puppet on mw183 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [05:36:38] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:36:40] @rhinosf1 I forgot to rebuild managewiki cache for all wikis so I'll do that but once thats done we should be good to go with launching the NextTide program. [05:39:07] CC @pskyechology ^ [05:39:50] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [05:39:50] !log [skye@test151] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/maintenance/createAndPromote.php --wiki=metawikibeta Skye --custom-groups=checkuser --force (END - exit=0) [05:39:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:40:22] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [05:41:05] !log [universalomega@mwtask181] finished deploy of {'l10n': True, 'versions': '1.44'} to all - SUCCESS in 454s [05:41:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:41:09] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:41:10] RECOVERY - mw202 Puppet on mw202 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:41:28] RECOVERY - mw203 Puppet on mw203 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [05:42:05] !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:ResetWikiCaches (START) [05:42:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:42:38] ^I have that running in a screen ... it'll take awhile [05:43:03] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:43:20] [1/2] I'll help the first 10 people interested on how to  start earning $100k or more within a week, but you will reimburse me 10% of your profits when you receive it. Note: only interested people should send a friend request or send me a dm! ask me (HOW) via Telegram username @Official_James01 [05:43:20] [2/2] Or The telegram link in my bio [05:43:48] RECOVERY - mwtask161 Puppet on mwtask161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:44:30] RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [05:44:33] RECOVERY - mw173 Puppet on mw173 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:47:03] RECOVERY - mw193 Puppet on mw193 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [05:47:18] lmao its still on 11 wikis this will take awhile lol [05:50:21] Check we've definitely tested everything important [05:50:41] The fact @pskyechology has just said 2FA is broken means it's not ready [05:50:53] Important security features are an issue [05:51:40] Like I said there I think its a composer issue with install not the extension but not certain. Yeah that's a must to have first. [05:51:40] might be a silly composer issue [05:52:25] !log [blankeclair@test151:/srv/mediawiki-staging/1.44/extensions/OATHAuth]$ sudo -u www-data composer update --no-dev [05:52:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:54:46] !log [blankeclair@test151:/srv/mediawiki-staging/1.44/extensions/OATHAuth]$ sudo -u www-data all_proxy=http://bastion.fsslc.wtnet:8080 composer update --no-dev [05:54:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:55:09] !log [blankeclair@test151] starting deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'OATHAuth'} to test151 [05:55:10] !log [blankeclair@test151] finished deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'OATHAuth'} to test151 - SUCCESS in 1s [05:55:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:55:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:04:05] !log [blankeclair@test151] starting deploy of {'folders': 'extensions/OATHAuth/vendor'} to test151 [06:04:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:04:15] !log [blankeclair@test151] starting deploy of {'folders': '1.44/extensions/OATHAuth/vendor'} to test151 [06:04:16] !log [blankeclair@test151] finished deploy of {'folders': '1.44/extensions/OATHAuth/vendor'} to test151 - SUCCESS in 0s [06:04:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:04:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:06:03] !log test151: sudo -u www-data rsync -vaP /srv/mediawiki{-staging,}/1.44/extensions/OATHAuth/vendor/ [06:06:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:09:10] !log [universalomega@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': ['CreateWiki', 'ManageWiki']} to all [06:09:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:10:01] !log [universalomega@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': ['CreateWiki', 'ManageWiki']} to all - SUCCESS in 50s [06:10:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:12:04] miraheze/ManageWiki - Universal-Omega the build has errored. [06:12:22] @rhinosf1 FYI thanks to the amazing shared virtual domains we can use upgrade.php if we decide to. [06:12:55] *update.php [06:13:08] miraheze/CreateWiki - Universal-Omega the build has errored. [06:13:51] ^ that was intentional and its fine for now. [06:21:54] for now [06:25:28] miraheze/mw-config - pskyechology the build passed. [06:27:23] it will fail until we do a full upgrade and I can merge in that other installertask patch lol [06:30:58] miraheze/mw-config - pskyechology the build passed. [06:31:08] !log [universalomega@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'DynamicPageList3'} to all [06:31:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:31:35] !log [universalomega@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'DynamicPageList3'} to all - SUCCESS in 26s [06:31:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:31:45] [02mw-config] 07pskyechology retitled PR #6025: "Add checkuser-temporary-account-auto-reveal to disallowed rights" ➜ "Disallow temporary account-related CheckUser right and group" 13https://github.com/miraheze/mw-config/pull/6025 [06:36:27] miraheze/mw-config - pskyechology the build passed. [06:36:40] [02mw-config] 07pskyechology retitled PR #6025: "Disallow temporary account-related CheckUser right and group" ➜ "Disallow temporary account-related CheckUser groups and right" 13https://github.com/miraheze/mw-config/pull/6025 [06:37:17] [02mw-config] 07Universal-Omega merged 07pskyechology's pull request #6025: Disallow temporary account-related CheckUser groups and right (07miraheze:03main...07pskyechology:03checkuser-temporary) 13https://github.com/miraheze/mw-config/pull/6025 [06:37:17] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/648336b818f3773cfe047b66e1360083e5300405 [06:37:18] 02mw-config/03main 07Skye 03648336b Disallow temporary account-related CheckUser groups and right (#6025)… [06:37:33] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [06:37:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:37:49] !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:ResetWikiCaches (END - exit=2) [06:37:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:37:58] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s [06:38:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:38:05] !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:ResetWikiCaches (START) [06:38:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:38:12] miraheze/mw-config - Universal-Omega the build passed. [06:56:52] !log [universalomega@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'DynamicPageList3'} to all [06:56:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:57:17] !log [universalomega@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'DynamicPageList3'} to all - SUCCESS in 25s [06:57:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:58:54] I decide not to [06:59:09] why? [06:59:11] I don't mind running it with --schema [06:59:26] Because then we have no visibility of what we're running [06:59:40] I'd rather we actually understand the schema changes and scripts [06:59:51] but then we miss so much [07:00:06] It has a --schema for dry running and you're fine on beta [07:00:14] But prod id rather was deliberate [07:00:35] It wouldn't be the first time upgrade.php deleted important tables and took entires wikis down [07:00:51] _looks at the time update.php was accidentally ran in wmf prod_ [07:01:57] https://wikitech.wikimedia.org/wiki/Incidents/2020-04-07_Wikidata%27s_wb_items_per_site_table_dropped [07:02:10] I ran it on just one wiki and it ran things we missed from 1.39 (the earliest the updater works) [07:02:55] (my own wiki though) [07:04:52] extreme caution should be taken [07:05:00] Running update.php in prod [07:05:07] There has been logic errors a fair few times [07:05:32] And you still need to think about migration variables [07:06:47] fair enough [07:06:55] as long as we dont miss anything. [07:07:28] We do have schema drift scripts [07:07:33] From @blankeclair [07:07:40] meow? [07:07:48] i forgot how they workedddddd [07:07:50] lol [07:08:04] also, i'm kinda sleepy [07:08:06] at 5 pm [07:08:09] Can you unforget? [07:08:18] And go hug Blahaj if sleepy [07:13:38] i am hugging blahaj, yes <3 [07:23:10] i rember [07:23:20] [1/2] https://github.com/Ladsgroup/db-analyzor-tools applied w/ this patch [07:23:20] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1394580075000889445/1752564199632.diff?ex=68775367&is=687601e7&hm=300e421549cfa34b206a02379e561ff2d38f40c0b074906ae8f4c987f9d33331& [07:23:20] [GitHub] Ladsgroup/db-analyzor-tools - Python tools to analyze databases, like finding drifts or abstracting sql | 1287.0% Python 1313.0% HTML | Last Push: 2024-12-04 - 15:33:55 UTC | Stargazers: 4 | Watchers: 1 | Forks: 4 | Network: 4 | Open Issues: 2 [07:25:10] `python -m pdb db_drift_checker.py core --prod --wiki testwiki 'sudo -u www-data /home/blankeclair/mariadb/usr/bin/mariadb --defaults-file=/srv/mediawiki/cache/CLAIRE-cognitohazard-my.cnf'` [07:26:29] claire cognitohazard O_O [07:26:46] it's a mariadb config file w/ the database password [07:27:14] damn i didnt know your password could instantly kill me [07:27:26] it's a cognitohazard because i didn't wanna see it :p [07:27:40] damn that ugly huh [07:27:55] idk if it's ugly [07:27:58] i have no idea how it looks lol [07:28:07] fair [07:32:26] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=forkedaltversewiki --new=allohistoricaversaliswiki --user=Reception123 (END - exit=256) [07:32:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:33:38] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=skibiditoiletwikiwiki --new=skibiditoiletwiki --user=Reception123 (END - exit=256) [07:33:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:36:08] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=accishiwiki --new=vehiclesofbrickrigswiki --user=Reception123 (END - exit=256) [07:36:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:08:07] The extension testing wiki [died again](https://exttest.mirabeta.org/wiki/Main_Page) thanks to LanguageSelector. I guess that's the end for today. [08:11:07] Also, it seems that [this wiki](https://nebu.miraheze.org/wiki/Szymon) is having issues with Wikimedia Commons. Some images that do exist on commons appear as red links. This [was originally reported on Meta's Help Desk](https://meta.miraheze.org/wiki/Help_desk#some_of_.svg_flags_are_broken,_and_some_aren't). [08:11:43] I tried purging and doing null edits, but they did not make a difference. [09:42:41] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49826MiB (10% inode=99%); [10:34:47] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49928MiB (10% inode=99%); [11:04:42] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49865MiB (10% inode=99%); [12:27:09] !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:ResetWikiCaches (END - exit=0) [12:27:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:19:17] !log [somerandomdeveloper@test151] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php ManageWiki:ToggleExtension --wiki=exttestwikibeta --name=languageselector --disable (END - exit=0) [13:19:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:37:23] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'TabberNeue'} to test151 [13:37:25] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'TabberNeue'} to test151 - SUCCESS in 2s [13:37:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:37:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:38:30] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'TabberNeue'} to all [13:38:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:39:03] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'TabberNeue'} to all - SUCCESS in 33s [13:39:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:18:38] Can you run it to see what patches 1.44 needs based on our 1.43 dbs? [14:58:14] Thank you! I'll see if I have some time to do more testing today. [16:40:10] !log upgrade phorge on phorge171 [16:40:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:04:19] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'LastModified'} to all [18:04:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:05:04] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'LastModified'} to all - SUCCESS in 44s [18:05:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:20:29] !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'LastModified'} to all [18:20:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:20:50] !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'LastModified'} to test151 [18:20:52] !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'LastModified'} to test151 - SUCCESS in 1s [18:20:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:20:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:21:13] !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'LastModified'} to all - SUCCESS in 44s [18:21:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:31:00] !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'Screenplay'} to test151 [19:31:02] !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'Screenplay'} to test151 - SUCCESS in 1s [19:31:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:31:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:53:36] once i can leave the gay shark station [20:56:00] Ok [20:56:52] !log [rhinos@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php CreateWiki:SetContainersAccess --wiki=issomtwiki (END - exit=0) [20:56:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:53:02] i found a bug in the schema diff script [23:31:23] [1/2] https://cdn.discordapp.com/attachments/808001911868489748/1394823693162516531/diff.diff?ex=6878364b&is=6876e4cb&hm=d59ddb6cafedce0295a511f4f1b27c76d4c293a38a162b899c2802193072c1b2& [23:31:23] [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1394823693561233418/drifts_core.json?ex=6878364b&is=6876e4cb&hm=c3defb555ae5f933ccbfc4a5fb81f0a6d493195215674ccbd84b1057a4189d17&