[00:07:18] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 57064MiB (12% inode=99%); [00:07:29] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 57217MiB (12% inode=99%); [00:08:03] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 57784MiB (12% inode=99%); [00:35:07] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 361 failures. Last run 1 minute ago with 361 failures. Failed resources (up to 3 shown): File[/srv/mediawiki-staging/1.44/extensions/3D],File[/srv/mediawiki-staging/1.44/extensions/AJAXPoll],File[/srv/mediawiki-staging/1.44/extensions/AbuseFilter],File[/srv/mediawiki-staging/1.44/extensions/AdminLinks] [00:45:12] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [00:51:29] !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/youtube.php /srv/mediawiki/1.43/maintenance/run.php ManageWiki:ToggleExtension --name=embedvideo (START) [00:51:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:38:35] !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/youtube.php /srv/mediawiki/1.43/maintenance/run.php ManageWiki:ToggleExtension --name=embedvideo (END - exit=0) [01:38:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:40:33] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php ManageWiki:ToggleExtension --wiki=metawiki --name=youtube --all-wikis --disable (END - exit=512) [01:40:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:40:52] miraheze/mw-config - Universal-Omega the build passed. [01:46:59] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php ManageWiki:ToggleExtension --wiki=metawiki --name=youtube --all-wikis --disable --execute (END - exit=0) [01:47:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:47:23] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [01:47:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:47:49] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s [01:47:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:46:08] PROBLEM - matomo151 Check unit status of matomo-archiver-3 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-3 [03:04:04] PROBLEM - puppet181 Disk Space on puppet181 is WARNING: DISK WARNING - free space: / 1995MiB (6% inode=80%); [03:08:04] RECOVERY - puppet181 Disk Space on puppet181 is OK: DISK OK - free space: / 4110MiB (13% inode=80%); [05:27:19] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.47, 18.90, 12.27 [05:41:19] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.30, 22.96, 19.56 [05:43:18] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 11.27, 19.30, 18.68 [06:13:19] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.59, 17.15, 14.68 [06:15:18] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 14.58, 16.29, 14.69 [08:02:08] RECOVERY - matomo151 Check unit status of matomo-archiver-3 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-3 [08:59:18] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49822MiB (10% inode=99%); [09:33:29] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49868MiB (10% inode=99%); [09:48:03] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49927MiB (10% inode=99%); [10:05:19] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.38, 19.20, 12.34 [10:39:28] !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.43/maintenance/run.php MediaModeration:importExistingFilesToScanTable (END - exit=0) [10:39:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:48:02] That's good [11:05:19] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 16.27, 20.80, 23.32 [11:13:19] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 16.29, 16.38, 20.06 [13:18:34] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [13:26:34] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [16:08:34] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [16:16:34] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [17:13:48] !log [somerandomdeveloper@test151] starting deploy of {'folders': '1.44/extensions/SemanticScribunto,1.44/extensions/SemanticDrilldown'} to test151 [17:13:49] !log [somerandomdeveloper@test151] finished deploy of {'folders': '1.44/extensions/SemanticScribunto,1.44/extensions/SemanticDrilldown'} to test151 - SUCCESS in 0s [17:13:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:13:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:16:04] !log [somerandomdeveloper@mwtask181] starting deploy of {'folders': '1.44/extensions/SemanticScribunto,1.44/extensions/SemanticDrilldown'} to all [17:16:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:16:53] !log [somerandomdeveloper@mwtask181] finished deploy of {'folders': '1.44/extensions/SemanticScribunto,1.44/extensions/SemanticDrilldown'} to all - SUCCESS in 49s [17:16:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:28:34] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [17:33:23] !log [universalomega@mwtask181] starting deploy of {'world': True, 'versions': '1.44'} to mwtask181 [17:33:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:34:17] !log [universalomega@mwtask181] finished deploy of {'world': True, 'versions': '1.44'} to mwtask181 - SUCCESS in 53s [17:34:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:36:34] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [17:48:34] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 26.59, 18.34, 10.90 [17:52:17] !log [somerandomdeveloper@test151] starting deploy of {'config': True} to test151 [17:52:18] !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [17:52:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:52:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:52:34] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.79, 9.92, 9.18 [17:53:18] [02mw-config] 07Universal-Omega created 03wb-fix (+1 new commit) 13https://github.com/miraheze/mw-config/commit/5d61796a3168 [17:53:18] 02mw-config/03wb-fix 07CosmicAlpha 035d61796 Wikibase: set defaultEntityNamespaces to false [17:53:23] [02mw-config] 07Universal-Omega opened pull request #6088: Wikibase: set defaultEntityNamespaces to false (03main...03wb-fix) 13https://github.com/miraheze/mw-config/pull/6088 [17:53:38] [02mw-config] 07Universal-Omega pushed 1 new commit to 03wb-fix 13https://github.com/miraheze/mw-config/commit/d2caf596e1fc417f29f3f4b04c28b1a6e7a11829 [17:53:38] 02mw-config/03wb-fix 07CosmicAlpha 03d2caf59 - [17:54:05] !log [somerandomdeveloper@test151] starting deploy of {'config': True} to test151 [17:54:06] !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [17:54:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:54:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:54:33] miraheze/mw-config - Universal-Omega the build passed. [17:55:08] !log [somerandomdeveloper@test151] starting deploy of {'config': True} to test151 [17:55:09] !log [somerandomdeveloper@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [17:55:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:55:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:59:44] [02mw-config] 07Universal-Omega merged pull request #6088: Wikibase: set defaultEntityNamespaces to false (03main...03wb-fix) 13https://github.com/miraheze/mw-config/pull/6088 [17:59:44] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/87e1175684f7fdefc8862dee4624da5ba789e08f [17:59:44] 02mw-config/03main 07CosmicAlpha 0387e1175 Wikibase: set defaultEntityNamespaces to false… [17:59:45] [02mw-config] 07Universal-Omega 04deleted 03wb-fix at 03d2caf59 13https://api.github.com/repos/miraheze/mw-config/commit/d2caf59 [17:59:58] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [18:00:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:00:24] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s [18:00:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:00:40] miraheze/mw-config - Universal-Omega the build passed. [18:09:40] !log [universalomega@mwtask181] starting deploy of {'world': True, 'versions': '1.44'} to mw151 [18:09:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:10:09] !log [universalomega@mwtask181] finished deploy of {'world': True, 'versions': '1.44'} to mw151 - SUCCESS in 28s [18:10:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:11:07] !log [universalomega@mwtask181] starting deploy of {'world': True, 'versions': '1.44'} to all [18:11:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:15:27] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 6.39, 12.60, 9.17 [18:17:25] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 1.66, 8.79, 8.18 [18:20:53] !log [universalomega@mwtask181] finished deploy of {'world': True, 'versions': '1.44'} to all - SUCCESS in 586s [18:20:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:21:18] !log [universalomega@mwtask181] starting deploy of {'world': True, 'force': True, 'versions': '1.44', 'ignore_time': True} to all [18:21:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:28:02] [02mw-config] 07Universal-Omega pushed 1 new commit to 03newwiki-mwversion 13https://github.com/miraheze/mw-config/commit/01b2be1d42a3cdd612a0bc6e15febc7262da3982 [18:28:02] 02mw-config/03newwiki-mwversion 07CosmicAlpha 0301b2be1 Fix [18:29:07] miraheze/mw-config - Universal-Omega the build passed. [18:29:41] !log [universalomega@mwtask181] finished deploy of {'world': True, 'force': True, 'versions': '1.44', 'ignore_time': True} to all - SUCCESS in 503s [18:29:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:30:30] [02mw-config] 07Universal-Omega pushed 1 new commit to 03newwiki-mwversion 13https://github.com/miraheze/mw-config/commit/0828bbf6b758533cd7028e1ff3a71b84d876e5ee [18:30:30] 02mw-config/03newwiki-mwversion 07CosmicAlpha 030828bbf Update [18:31:21] miraheze/mw-config - Universal-Omega the build passed. [18:36:44] [02puppet] 07Universal-Omega created 03cwai-dryrun (+1 new commit) 13https://github.com/miraheze/puppet/commit/e8631c4d6951 [18:36:44] 02puppet/03cwai-dryrun 07CosmicAlpha 03e8631c4 Set CreateWiki AI to dryrun [18:36:50] [02puppet] 07Universal-Omega opened pull request #4511: Set CreateWiki AI to dryrun (03main...03cwai-dryrun) 13https://github.com/miraheze/puppet/pull/4511 [18:36:56] [02puppet] 07coderabbitai[bot] commented on pull request #4511:
[…] 13https://github.com/miraheze/puppet/pull/4511#issuecomment-3263962477 [18:37:44] [02puppet] 07Universal-Omega merged pull request #4511: Set CreateWiki AI to dryrun (03main...03cwai-dryrun) 13https://github.com/miraheze/puppet/pull/4511 [18:37:44] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/cc2d30711c23dcf1d54e2d52d049e2090d922eda [18:37:45] 02puppet/03main 07CosmicAlpha 03cc2d307 Set CreateWiki AI to dryrun [18:37:45] [02puppet] 07Universal-Omega 04deleted 03cwai-dryrun at 03e8631c4 13https://api.github.com/repos/miraheze/puppet/commit/e8631c4 [18:38:34] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [18:41:27] [02mw-config] 07Universal-Omega merged pull request #6087: Set MediaWiki version for all new wikis to MediaWiki 1.44 (03main...03newwiki-mwversion) 13https://github.com/miraheze/mw-config/pull/6087 [18:41:27] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/7190cf6d3895c7851f9dd4f0638d401bb9c57cbb [18:41:27] 02mw-config/03main 07CosmicAlpha 037190cf6 Set MediaWiki version for all new wikis to MediaWiki 1.44 [18:41:28] [02mw-config] 07Universal-Omega 04deleted 03newwiki-mwversion at 030828bbf 13https://api.github.com/repos/miraheze/mw-config/commit/0828bbf [18:41:35] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [18:41:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:42:09] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 34s [18:42:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:42:20] miraheze/mw-config - Universal-Omega the build passed. [18:46:34] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [18:47:16] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/49d8a6b3458f1daa61e3b8e2caf5cb71caaa2b6e [18:47:16] 02mw-config/03main 07CosmicAlpha 0349d8a6b Fix [18:47:44] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [18:47:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:48:13] miraheze/mw-config - Universal-Omega the build passed. [18:48:16] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 31s [18:48:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:59:53] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/b5cc72ecc3cccfd187aa2c54134116dd6e7d8d5c [18:59:53] 02mw-config/03main 07CosmicAlpha 03b5cc72e Fix [18:59:59] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [19:00:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:00:23] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 23s [19:00:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:00:51] miraheze/mw-config - Universal-Omega the build passed. [19:06:46] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/e92e987d5a57e96ddc5383018aa0100b341d1970 [19:06:47] 02mw-config/03main 07CosmicAlpha 03e92e987 Revert [19:07:02] [02mw-config] 07Universal-Omega created 03revert-6087-newwiki-mwversion (+1 new commit) 13https://github.com/miraheze/mw-config/commit/72a56ce83b98 [19:07:02] 02mw-config/03revert-6087-newwiki-mwversion 07CosmicAlpha 0372a56ce Revert "Set MediaWiki version for all new wikis to MediaWiki 1.44"… [19:07:06] [02mw-config] 07Universal-Omega opened pull request #6089: Revert "Set MediaWiki version for all new wikis to MediaWiki 1.44" (03main...03revert-6087-newwiki-mwversion) 13https://github.com/miraheze/mw-config/pull/6089 [19:07:51] miraheze/mw-config - Universal-Omega the build passed. [19:07:59] [02mw-config] 07Universal-Omega merged pull request #6089: Revert "Set MediaWiki version for all new wikis to MediaWiki 1.44" (03main...03revert-6087-newwiki-mwversion) 13https://github.com/miraheze/mw-config/pull/6089 [19:07:59] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/31c243539e8d3cca1f5ff0dff82c6be521004d24 [19:07:59] 02mw-config/03main 07CosmicAlpha 0331c2435 Revert "Set MediaWiki version for all new wikis to MediaWiki 1.44" (#6089)… [19:08:00] miraheze/mw-config - Universal-Omega the build passed. [19:08:01] [02mw-config] 07Universal-Omega 04deleted 03revert-6087-newwiki-mwversion at 0372a56ce 13https://api.github.com/repos/miraheze/mw-config/commit/72a56ce [19:08:03] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [19:08:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:28] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 24s [19:08:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:08:54] miraheze/mw-config - Universal-Omega the build passed. [19:14:17] [02MirahezeMagic] 07Universal-Omega created 03Populate143MediaWikiVersion (+1 new commit) 13https://github.com/miraheze/MirahezeMagic/commit/3bd5baf37310 [19:14:17] 02MirahezeMagic/03Populate143MediaWikiVersion 07CosmicAlpha 033bd5baf Add Populate143MediaWikiVersion script [19:14:20] [02MirahezeMagic] 07Universal-Omega opened pull request #594: Add Populate143MediaWikiVersion script (03main...03Populate143MediaWikiVersion) 13https://github.com/miraheze/MirahezeMagic/pull/594 [19:14:27] [02MirahezeMagic] 07coderabbitai[bot] commented on pull request #594:
[…] 13https://github.com/miraheze/MirahezeMagic/pull/594#issuecomment-3263983109 [19:16:51] [02MirahezeMagic] 07Universal-Omega merged pull request #594: Add Populate143MediaWikiVersion script (03main...03Populate143MediaWikiVersion) 13https://github.com/miraheze/MirahezeMagic/pull/594 [19:16:51] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/15351a319f174995924e70eb21d2fd12bf10bdb8 [19:16:51] 02MirahezeMagic/03main 07CosmicAlpha 0315351a3 Add Populate143MediaWikiVersion script [19:16:52] [02MirahezeMagic] 07Universal-Omega 04deleted 03Populate143MediaWikiVersion at 033bd5baf 13https://api.github.com/repos/miraheze/MirahezeMagic/commit/3bd5baf [19:17:13] !log [universalomega@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'MirahezeMagic'} to all [19:17:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:01] !log [universalomega@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 48s [19:18:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:34] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [19:20:53] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:Populate143MediaWikiVersion --wiki=metawiki --dry-run (END - exit=2) [19:20:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:24:01] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/cba8f1cc53cfb9790e726111a884fb75ede164ca [19:24:01] 02MirahezeMagic/03main 07CosmicAlpha 03cba8f1c Fix [19:24:06] !log [universalomega@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'MirahezeMagic'} to all [19:24:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:24:53] !log [universalomega@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 47s [19:24:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:13] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:Populate143MediaWikiVersion --wiki=metawiki --dry-run (END - exit=2) [19:26:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:34] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [19:26:37] miraheze/MirahezeMagic - Universal-Omega the build passed. [19:30:12] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/e2d2b4c06a2405f0a7607081e4bb7cc108eb4088 [19:30:12] 02MirahezeMagic/03main 07CosmicAlpha 03e2d2b4c Fix [19:30:24] !log [universalomega@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'MirahezeMagic'} to all [19:30:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:31:10] !log [universalomega@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 46s [19:31:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:05] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:Populate143MediaWikiVersion --wiki=metawiki --dry-run (END - exit=2) [19:37:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:40:34] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 12.91, 8.03, 3.76 [19:42:08] miraheze/MirahezeMagic - Universal-Omega the build passed. [19:42:34] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 2.18, 5.55, 3.37 [19:58:34] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [20:06:34] RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push [20:20:07] [02mw-config] 07Universal-Omega 04force-pushed 03cs_associated_pages from 032bd6560 to 036279ca2 13https://api.github.com/repos/miraheze/mw-config/commits/cs_associated_pages [20:20:07] 02mw-config/03cs_associated_pages 07CosmicAlpha 036279ca2 Add cs_associated_pages to CommentStreams SQL [20:21:01] miraheze/mw-config - Universal-Omega the build passed. [20:21:57] [02mw-config] 07Universal-Omega pushed 1 new commit to 03cs_associated_pages 13https://github.com/miraheze/mw-config/commit/0ac597bb159fd6714a32673cbdc72ffc8f3f5000 [20:21:58] 02mw-config/03cs_associated_pages 07CosmicAlpha 030ac597b Only on 1.44 [20:22:09] [02mw-config] 07Universal-Omega retitled PR #6030: "Add cs_associated_pages to CommentStreams SQL" ➜ "Add cs_associated_pages to CommentStreams SQL on 1.44" 13https://github.com/miraheze/mw-config/pull/6030 [20:22:51] miraheze/mw-config - Universal-Omega the build passed. [20:24:48] [02mw-config] 07Universal-Omega merged pull request #6030: Add cs_associated_pages to CommentStreams SQL on 1.44 (03main...03cs_associated_pages) 13https://github.com/miraheze/mw-config/pull/6030 [20:24:48] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/c37afee00ab938527b1268cb0dd4cfc5aef5334a [20:24:48] 02mw-config/03main 07CosmicAlpha 03c37afee Add cs_associated_pages to CommentStreams SQL on 1.44 [20:24:50] [02mw-config] 07Universal-Omega 04deleted 03cs_associated_pages at 030ac597b 13https://api.github.com/repos/miraheze/mw-config/commit/0ac597b [20:25:41] miraheze/mw-config - Universal-Omega the build passed. [20:25:44] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [20:25:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:26:10] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s [20:26:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:29:04] [02mw-config] 07Universal-Omega created 031.44-stable (+1 new commit) 13https://github.com/miraheze/mw-config/commit/0dd0acc48edd [20:29:04] 02mw-config/031.44-stable 07CosmicAlpha 030dd0acc Update stable version to 1.44 and make 1.43 legacy [20:29:09] [02mw-config] 07Universal-Omega opened pull request #6090: Update stable version to 1.44 and make 1.43 legacy (03main...031.44-stable) 13https://github.com/miraheze/mw-config/pull/6090 [20:30:01] miraheze/mw-config - Universal-Omega the build passed. [20:38:19] [02mw-config] 07Universal-Omega created 03remove-growthexperiments from 03main (+0 new commit) 13https://github.com/miraheze/mw-config/compare/remove-growthexperiments [20:38:50] [02mw-config] 07Universal-Omega pushed 1 new commit to 03remove-growthexperiments 13https://github.com/miraheze/mw-config/commit/c4710425c67b3fcda54bf987e88f61b9c4dfe49d [20:38:50] 02mw-config/03remove-growthexperiments 07CosmicAlpha 03c471042 Remove GrowthExperiments [20:39:03] [02mw-config] 07Universal-Omega opened pull request #6091: Remove GrowthExperiments (03main...03remove-growthexperiments) 13https://github.com/miraheze/mw-config/pull/6091 [20:39:57] miraheze/mw-config - Universal-Omega the build passed. [20:50:45] !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php updateSpecialPages --wiki=falcomwiki (END - exit=0) [20:50:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:10:25] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.43/maintenance/run.php MirahezeMagic:Populate143MediaWikiVersion --wiki=metawiki (END - exit=0) [21:10:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:11:07] !log backing up mhglobal, metawiki, and loginwiki [21:11:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:17:30] [02mw-config] 07Universal-Omega merged pull request #6090: Update stable version to 1.44 and make 1.43 legacy (03main...031.44-stable) 13https://github.com/miraheze/mw-config/pull/6090 [21:17:32] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/507415aec0c00a2d5987c866c26a98d773941a57 [21:17:34] [02mw-config] 07Universal-Omega 04deleted 031.44-stable at 030dd0acc 13https://api.github.com/repos/miraheze/mw-config/commit/0dd0acc [21:17:35] 02mw-config/03main 07CosmicAlpha 03507415a Update stable version to 1.44 and make 1.43 legacy [21:18:30] miraheze/mw-config - Universal-Omega the build passed. [21:45:56] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/8bb7331be493d3ad782b3e8b3ba83bc38863c00b [21:45:56] 02ssl/03main 07WikiTideBot 038bb7331 Bot: Auto-update domain lists [21:58:00] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [21:58:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:58:23] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s [21:58:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:59:52] [02puppet] 07Universal-Omega created 03revert-4511-cwai-dryrun (+1 new commit) 13https://github.com/miraheze/puppet/commit/7ba02b02df40 [21:59:52] 02puppet/03revert-4511-cwai-dryrun 07CosmicAlpha 037ba02b0 Revert "Set CreateWiki AI to dryrun"… [22:00:02] [02puppet] 07Universal-Omega opened pull request #4512: Revert "Set CreateWiki AI to dryrun" (03main...03revert-4511-cwai-dryrun) 13https://github.com/miraheze/puppet/pull/4512 [22:02:39] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/26d5e6f4552d834710a02476b08ac4ba6c3e9b35 [22:02:39] 02mw-config/03main 07CosmicAlpha 0326d5e6f Always use 1.44 in CreateWikiSQLFiles [22:02:53] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [22:02:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:03:15] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s [22:03:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:03:35] miraheze/mw-config - Universal-Omega the build passed. [22:03:48] [02puppet] 07Universal-Omega merged pull request #4512: Revert "Set CreateWiki AI to dryrun" (03main...03revert-4511-cwai-dryrun) 13https://github.com/miraheze/puppet/pull/4512 [22:03:50] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/3f05face3e2cb7561a2e7333895013e46fdd06dc [22:03:52] [02puppet] 07Universal-Omega 04deleted 03revert-4511-cwai-dryrun at 037ba02b0 13https://api.github.com/repos/miraheze/puppet/commit/7ba02b0 [22:03:53] 02puppet/03main 07CosmicAlpha 033f05fac Revert "Set CreateWiki AI to dryrun"… [22:17:50] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php ManageWiki:ToggleExtension --wiki=metawiki --name=growthexperiments --all-wikis --disable --execute (END - exit=0) [22:17:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:18:01] [02mw-config] 07Universal-Omega merged pull request #6091: Remove GrowthExperiments (03main...03remove-growthexperiments) 13https://github.com/miraheze/mw-config/pull/6091 [22:18:01] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/0bcd41b32bd9ae7c05187db51d02e85e7cea47db [22:18:01] 02mw-config/03main 07CosmicAlpha 030bcd41b Remove GrowthExperiments [22:18:03] [02mw-config] 07Universal-Omega 04deleted 03remove-growthexperiments at 03c471042 13https://api.github.com/repos/miraheze/mw-config/commit/c471042 [22:18:08] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [22:18:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:18:31] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 23s [22:18:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:18:57] miraheze/mw-config - Universal-Omega the build passed. [22:19:37] [02mediawiki-repos] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mediawiki-repos/commit/401db4fd6d10e27c5d36d8be0455f24ff94c6063 [22:19:37] 02mediawiki-repos/03main 07CosmicAlpha 03401db4f Remove GrowthExperiments [22:20:09] [02puppet] 07coderabbitai[bot] commented on pull request #4512:
[…] 13https://github.com/miraheze/puppet/pull/4512#issuecomment-3264108670 [22:33:39] [02puppet] 07Universal-Omega created 031.44-default (+1 new commit) 13https://github.com/miraheze/puppet/commit/9e8448960030 [22:33:40] 02puppet/031.44-default 07CosmicAlpha 039e84489 Make 1.44 the default version [22:33:43] [02puppet] 07Universal-Omega opened pull request #4513: Make 1.44 the default version (03main...031.44-default) 13https://github.com/miraheze/puppet/pull/4513 [22:33:50] [02puppet] 07coderabbitai[bot] commented on pull request #4513:
[…] 13https://github.com/miraheze/puppet/pull/4513#issuecomment-3264115542 [22:34:51] [02puppet] 07Universal-Omega merged pull request #4513: Make 1.44 the default version (03main...031.44-default) 13https://github.com/miraheze/puppet/pull/4513 [22:34:53] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/a357bb8c21a3ba225e684cf82123fda25a28354c [22:34:55] [02puppet] 07Universal-Omega 04deleted 031.44-default at 039e84489 13https://api.github.com/repos/miraheze/puppet/commit/9e84489 [22:34:56] 02puppet/03main 07CosmicAlpha 03a357bb8 Make 1.44 the default version [22:37:10] [02puppet] 07Universal-Omega created 03beta-1.45 (+1 new commit) 13https://github.com/miraheze/puppet/commit/bf05c75879c5 [22:37:10] 02puppet/03beta-1.45 07CosmicAlpha 03bf05c75 beta: add 1.45 and remove 1.43 [22:37:16] [02puppet] 07Universal-Omega opened pull request #4514: beta: add 1.45 and remove 1.43 (03main...03beta-1.45) 13https://github.com/miraheze/puppet/pull/4514 [22:37:22] [02puppet] 07coderabbitai[bot] commented on pull request #4514:
[…] 13https://github.com/miraheze/puppet/pull/4514#issuecomment-3264117369 [22:38:51] [02mediawiki-repos] 07Universal-Omega created 03dpl4-1.45 (+1 new commit) 13https://github.com/miraheze/mediawiki-repos/commit/fb80ad7470c5 [22:38:51] 02mediawiki-repos/03dpl4-1.45 07CosmicAlpha 03fb80ad7 Add DynamicPageList4 on MediaWiki 1.45 [22:38:55] [02mediawiki-repos] 07Universal-Omega opened pull request #91: Add DynamicPageList4 on MediaWiki 1.45 (03main...03dpl4-1.45) 13https://github.com/miraheze/mediawiki-repos/pull/91 [22:39:01] [02mediawiki-repos] 07coderabbitai[bot] commented on pull request #91:
[…] 13https://github.com/miraheze/mediawiki-repos/pull/91#issuecomment-3264118185 [23:24:16] [02mediawiki-repos] 07Universal-Omega merged pull request #91: Add DynamicPageList4 on MediaWiki 1.45 (03main...03dpl4-1.45) 13https://github.com/miraheze/mediawiki-repos/pull/91 [23:24:17] [02mediawiki-repos] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mediawiki-repos/commit/5b55c2a155474fa31170af6f452ebc639c815788 [23:24:17] 02mediawiki-repos/03main 07CosmicAlpha 035b55c2a Add DynamicPageList4 on MediaWiki 1.45 [23:24:17] [02mediawiki-repos] 07Universal-Omega 04deleted 03dpl4-1.45 at 03fb80ad7 13https://api.github.com/repos/miraheze/mediawiki-repos/commit/fb80ad7 [23:24:48] [02puppet] 07Universal-Omega merged pull request #4514: beta: add 1.45 and remove 1.43 (03main...03beta-1.45) 13https://github.com/miraheze/puppet/pull/4514 [23:24:49] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/bafc533b3e1153e3d24ac0c2f36207dcd3c014af [23:24:49] 02puppet/03main 07CosmicAlpha 03bafc533 beta: add 1.45 and remove 1.43 [23:24:50] [02puppet] 07Universal-Omega 04deleted 03beta-1.45 at 03bf05c75 13https://api.github.com/repos/miraheze/puppet/commit/bf05c75 [23:33:01] [02mw-config] 07Universal-Omega created 03dblists (+1 new commit) 13https://github.com/miraheze/mw-config/commit/b18f94d4fffb [23:33:01] 02mw-config/03dblists 07CosmicAlpha 03b18f94d Add closed and inactive database lists [23:33:09] [02mw-config] 07Universal-Omega opened pull request #6092: Add closed and inactive database lists (03main...03dblists) 13https://github.com/miraheze/mw-config/pull/6092 [23:34:06] miraheze/mw-config - Universal-Omega the build passed. [23:34:58] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/98fd29ee836279a564ca1f224e94bc48d1557c4b [23:35:00] 02puppet/03main 07CosmicAlpha 0398fd29e mwscript: add closed and inactive to validDBLists [23:35:23] [02mw-config] 07Universal-Omega merged pull request #6092: Add closed and inactive database lists (03main...03dblists) 13https://github.com/miraheze/mw-config/pull/6092 [23:35:23] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/81f182c4b4f82b70e3e4a2ae639c88dc1c44e17c [23:35:24] 02mw-config/03main 07CosmicAlpha 0381f182c Add closed and inactive database lists [23:35:25] [02mw-config] 07Universal-Omega 04deleted 03dblists at 03b18f94d 13https://api.github.com/repos/miraheze/mw-config/commit/b18f94d [23:36:12] miraheze/puppet - Universal-Omega the build has errored. [23:36:18] miraheze/mw-config - Universal-Omega the build passed. [23:36:39] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [23:36:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:37:02] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s [23:37:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:43:52] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_clone_femiwiki-deploy-1.45] [23:45:33] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/71bf06630eac8fb6a41614687e8919661f58edf0 [23:45:33] 02MirahezeMagic/03main 07CosmicAlpha 0371bf066 Fix [23:45:42] !log [universalomega@mwtask181] starting deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'MirahezeMagic'} to all [23:45:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:46:25] !log [universalomega@mwtask181] finished deploy of {'versions': ['1.43', '1.44'], 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 43s [23:46:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:46:53] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 23.80, 16.49, 7.39 [23:49:41] PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [23:49:59] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:50:03] PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [23:50:15] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [23:50:17] PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 14 backends are down. mw151 mw152 mw161 mw171 mw172 mw181 mw182 mw163 mw173 mw191 mw192 mw201 mw202 mw203 [23:50:26] PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 15 backends are down. mw151 mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw163 mw173 mw191 mw192 mw202 mw203 [23:50:40] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 13 backends are down. mw152 mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw173 mw191 mw192 mw193 mw203 [23:50:40] PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 12 backends are down. mw161 mw162 mw171 mw172 mw181 mw182 mw153 mw183 mw191 mw192 mw193 mw203 [23:52:16] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4161 bytes in 4.615 second response time [23:52:21] PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 49% [23:54:06] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.033 second response time [23:55:00] PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [23:56:27] PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [23:57:41] miraheze/MirahezeMagic - Universal-Omega the build passed. [23:58:34] PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push [23:59:04] RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4216 bytes in 0.968 second response time [23:59:42] RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4161 bytes in 0.078 second response time [23:59:45] RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4161 bytes in 0.074 second response time [23:59:51] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy