[00:00:20] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [00:00:26] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08993735909 secs [00:02:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.09, 3.79, 3.73 [00:03:08] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 11.51, 7.18, 4.03 [00:03:45] PROBLEM - cp33 Disk Space on cp33 is WARNING: DISK WARNING - free space: / 7852 MB (10% inode=98%); [00:04:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.13, 3.50, 3.63 [00:05:45] RECOVERY - cp33 Disk Space on cp33 is OK: DISK OK - free space: / 14269 MB (19% inode=98%); [00:06:44] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 1.31, 2.74, 3.78 [00:07:29] PROBLEM - matomo131 HTTPS on matomo131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:08:18] !log [universalomega@mwtask141] starting deploy of {'folders': 'w/extensions/IPInfo'} to all [00:08:34] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.91, 3.78, 3.70 [00:08:45] !log [universalomega@mwtask141] finished deploy of {'folders': 'w/extensions/IPInfo'} to all - SUCCESS in 26s [00:09:01] !log [universalomega@mwtask141] starting deploy of {'folders': 'w/extensions/Translate'} to all [00:09:29] !log [universalomega@mwtask141] finished deploy of {'folders': 'w/extensions/Translate'} to all - SUCCESS in 28s [00:10:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.36, 3.11, 3.46 [00:10:44] RECOVERY - matomo131 Current Load on matomo131 is OK: OK - load average: 1.38, 2.24, 3.34 [00:11:32] RECOVERY - matomo131 HTTPS on matomo131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.178 second response time [00:14:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.61, 3.56, 3.54 [00:16:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.82, 3.96, 3.71 [00:17:21] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 3.68, 4.72, 2.80 [00:18:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.24, 4.47, 3.95 [00:19:21] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.18, 3.45, 2.56 [00:20:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.90, 3.88, 3.80 [00:21:21] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.35, 2.67, 2.38 [00:24:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.82, 3.96, 3.80 [00:25:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e2fca31f6bb8...06a568b39cee [00:25:19] [02miraheze/mw-config] 07Universal-Omega 0306a568b - Apply discussiontools_persistent.sql with DiscussionTools install [00:25:25] PROBLEM - test131 NTP time on test131 is CRITICAL: NTP CRITICAL: Offset 0.73231861 secs [00:25:57] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [00:26:04] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [00:26:10] !log [universalomega@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/discussiontools.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/DiscussionTools/sql/mysql/discussiontools_persistent.sql (START) [00:26:19] miraheze/mw-config - Universal-Omega the build passed. [00:26:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.78, 3.68, 3.73 [00:27:28] RECOVERY - test131 NTP time on test131 is OK: NTP OK: Offset -0.0004217922688 secs [00:28:23] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 3.19, 3.43, 3.37 [00:30:19] RECOVERY - matomo131 Current Load on matomo131 is OK: OK - load average: 3.35, 3.28, 3.32 [00:30:33] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.46, 2.72, 3.35 [00:31:54] !log [universalomega@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/discussiontools.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/DiscussionTools/sql/mysql/discussiontools_persistent.sql (END - exit=0) [00:34:14] PROBLEM - matomo131 Current Load on matomo131 is CRITICAL: CRITICAL - load average: 4.30, 3.99, 3.61 [00:36:10] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 2.51, 3.45, 3.45 [00:36:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.22, 3.43, 3.48 [00:38:06] RECOVERY - matomo131 Current Load on matomo131 is OK: OK - load average: 2.15, 3.05, 3.31 [00:38:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.89, 4.04, 3.71 [00:40:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.30, 3.41, 3.52 [00:41:39] PROBLEM - es141 Current Load on es141 is CRITICAL: CRITICAL - load average: 5.20, 3.59, 2.71 [00:42:33] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.85, 2.99, 3.35 [00:43:57] PROBLEM - matomo131 Current Load on matomo131 is CRITICAL: CRITICAL - load average: 4.20, 3.82, 3.58 [00:45:39] RECOVERY - es141 Current Load on es141 is OK: OK - load average: 2.24, 3.21, 2.80 [00:51:29] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.77, 3.95, 3.55 [00:53:01] [02puppet] 07AgentIsai opened pull request 03#3133: Get more IP databases - 13https://github.com/miraheze/puppet/pull/3133 [00:53:25] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.03, 3.45, 3.41 [00:54:17] [02puppet] 07AgentIsai synchronize pull request 03#3133: Get more IP databases - 13https://github.com/miraheze/puppet/pull/3133 [00:55:22] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.19, 3.36, 3.38 [00:57:08] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.33, 6.39, 7.89 [00:59:29] [02mediawiki] 07dependabot[bot] opened pull request 03#12832: Bump extensions/Mermaid from `09186a9` to `74e06fb` - 13https://github.com/miraheze/mediawiki/pull/12832 [00:59:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_39/extensions/Mermaid-74e06fb [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/021be0948a99 [00:59:33] [02miraheze/mediawiki] 07dependabot[bot] 03021be09 - Bump extensions/Mermaid from `09186a9` to `74e06fb` [00:59:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_39/extensions/Mermaid-74e06fb - 13https://github.com/miraheze/mediawiki [00:59:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#12832: Bump extensions/Mermaid from `09186a9` to `74e06fb` - 13https://github.com/miraheze/mediawiki/pull/12832 [00:59:47] [02mediawiki] 07github-actions[bot] labeled pull request 03#12832: Bump extensions/Mermaid from `09186a9` to `74e06fb` - 13https://github.com/miraheze/mediawiki/pull/12832 [01:01:42] PROBLEM - es141 Current Load on es141 is WARNING: WARNING - load average: 3.42, 3.07, 2.74 [01:03:42] RECOVERY - es141 Current Load on es141 is OK: OK - load average: 2.85, 3.00, 2.76 [01:07:32] [02puppet] 07AgentIsai synchronize pull request 03#3133: Get more IP databases - 13https://github.com/miraheze/puppet/pull/3133 [01:09:08] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.94, 4.84, 6.47 [01:21:08] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 6.85, 6.01, 5.96 [01:23:08] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 8.65, 6.38, 6.07 [01:25:08] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.44, 5.32, 5.72 [01:27:28] !log [universalomega@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/maintenance/archives/patch-templatelinks-tl_title-nullable.sql (END - exit=0) [01:28:00] !log [universalomega@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/maintenance/archives/patch-templatelinks-drop-tl_title.sql (START) [01:31:08] RECOVERY - mwtask141 Disk Space on mwtask141 is OK: DISK OK - free space: / 12808 MB (17% inode=94%); [01:31:30] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.48, 3.59, 3.32 [01:32:25] PROBLEM - mwtask141 Puppet on mwtask141 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 8 minutes ago with 0 failures [01:34:37] PROBLEM - es141 PowerDNS Recursor on es141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:34:56] !log recloning /srv/mediawiki-staging/w on mwtask141 [01:35:41] PROBLEM - es141 Current Load on es141 is CRITICAL: CRITICAL - load average: 4.19, 3.84, 3.14 [01:36:21] !log restart logbot on mon141 [01:36:34] RECOVERY - es141 PowerDNS Recursor on es141 is OK: DNS OK: 1.858 second response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [01:37:03] !log restart logbot on mon141 [01:37:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.19, 3.30, 3.35 [01:37:36] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:37:41] PROBLEM - es141 Current Load on es141 is WARNING: WARNING - load average: 2.69, 3.42, 3.07 [01:39:40] RECOVERY - es141 Current Load on es141 is OK: OK - load average: 2.16, 3.01, 2.96 [01:40:05] !log restart logbot on mon141 [01:42:59] !log restart logbot on mon141 [01:43:08] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 8.81, 6.75, 5.73 [01:43:12] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.87, 5.04, 3.99 [01:45:08] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 4.68, 5.73, 5.47 [01:52:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.64, 3.80, 3.86 [01:56:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.89, 3.24, 3.89 [01:58:28] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.38, 3.52, 3.90 [02:00:21] !log restart logbot on mon141 [02:00:22] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:00:42] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.91, 2.53, 3.29 [02:04:01] Maybe the mediawiki internal error screen should have Miraheze replacement [02:05:36] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:05:39] ughh.... logbot!! [02:06:13] Naleksuh: maybe if we knew how or if it was an issue we'd prioritise looking into but it isn't. [02:06:48] cosmicalpha : But if you did, then you could just fix it. Instead, this is for when it isn't an issue [02:09:05] Fix it? I am trying to fix the issue. [02:09:24] I am confused. [02:11:26] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.43, 3.66, 3.48 [02:13:23] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.00, 3.06, 3.29 [02:15:08] PROBLEM - mwtask141 Disk Space on mwtask141 is WARNING: DISK WARNING - free space: / 7530 MB (10% inode=91%); [02:16:50] Cosmicalpha : I meant for the miraheze replacement as a longterm ting not just for now [02:17:06] So, that's why I was saying that fixing the issue vs. adding a custom error are different tasks [02:17:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.21, 3.67, 3.47 [02:18:57] !log [universalomega@mwtask141] starting deploy of {'folders': 'w/vendor'} to all [02:19:16] !log [universalomega@mwtask141] finished deploy of {'folders': 'w/vendor'} to all - SUCCESS in 19s [02:20:17] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[ipinfo_composer],Exec[translate_composer] [02:21:12] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.37, 3.40, 3.45 [02:22:53] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/78b37d4a4625...642e523fbe63 [02:22:55] [02miraheze/puppet] 07Universal-Omega 03642e523 - mediawiki: use http_proxy for composer install [02:23:09] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.08, 3.78, 3.58 [02:32:51] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.34, 3.41, 3.61 [02:34:48] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.70, 2.90, 3.40 [02:39:21] [02miraheze/CreateWiki] 07Universal-Omega pushed 0385 commits to 03Universal-Omega-patch-4 [+12/-0/±234] 13https://github.com/miraheze/CreateWiki/compare/26b87a23c35a...56fd2367523b [02:39:22] [02miraheze/CreateWiki] 07Universal-Omega 0356fd236 - Merge branch 'master' into Universal-Omega-patch-4 [02:39:24] [02CreateWiki] 07Universal-Omega synchronize pull request 03#331: Actually do update on changes being commited - 13https://github.com/miraheze/CreateWiki/pull/331 [02:42:26] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [02:42:41] miraheze/CreateWiki - Universal-Omega the build has errored. [02:47:56] !log [universalomega@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/maintenance/archives/patch-templatelinks-drop-tl_title.sql (END - exit=0) [02:56:17] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.02, 4.71, 3.65 [02:58:32] !log restart logbot on mon141 [03:07:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.27, 3.39, 3.74 [03:13:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.98, 2.50, 3.24 [03:23:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/40fbe65c8287 [03:23:31] [02miraheze/mw-config] 07Universal-Omega 0340fbe65 - Don't set DBO_DEFAULT flag [03:23:34] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [03:23:37] [02mw-config] 07Universal-Omega opened pull request 03#5085: Don't set DBO_DEFAULT flag - 13https://github.com/miraheze/mw-config/pull/5085 [03:24:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/40fbe65c8287...eca33b957314 [03:24:10] [02miraheze/mw-config] 07Universal-Omega 03eca33b9 - Update GlobalCache.php [03:24:13] [02mw-config] 07Universal-Omega synchronize pull request 03#5085: Don't set DBO_DEFAULT flag - 13https://github.com/miraheze/mw-config/pull/5085 [03:24:35] miraheze/mw-config - Universal-Omega the build passed. [03:25:15] miraheze/mw-config - Universal-Omega the build passed. [03:28:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/5ce4e9e46fdf [03:28:30] [02miraheze/mw-config] 07Universal-Omega 035ce4e9e - Restrict SlackNotifications and PdfBook [03:28:31] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-5 - 13https://github.com/miraheze/mw-config [03:28:32] [02mw-config] 07Universal-Omega opened pull request 03#5086: Restrict SlackNotifications and PdfBook - 13https://github.com/miraheze/mw-config/pull/5086 [03:29:38] miraheze/mw-config - Universal-Omega the build passed. [03:30:16] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/739bcd8aee5a...d27583f91746 [03:30:19] [02miraheze/mediawiki] 07Universal-Omega 03d27583f - Update Pivot and MirahezeMagic to master [03:30:33] !log [void@mon141] test [03:30:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/06a568b39cee...8400652fd19f [03:30:58] [02miraheze/mw-config] 07Universal-Omega 038400652 - Restrict SlackNotifications and PdfBook (#5086) [03:30:59] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 [03:31:00] [02mw-config] 07Universal-Omega closed pull request 03#5086: Restrict SlackNotifications and PdfBook - 13https://github.com/miraheze/mw-config/pull/5086 [03:31:01] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 - 13https://github.com/miraheze/mw-config [03:31:58] miraheze/mw-config - Universal-Omega the build passed. [03:34:03] [02mediawiki] 07Universal-Omega commented on pull request 03#12680: Bump extensions/MirahezeMagic from `972c0f2` to `5cf7a35` - 13https://github.com/miraheze/mediawiki/pull/12680#issuecomment-1399752840 [03:34:05] [02mediawiki] 07dependabot[bot] edited pull request 03#12680: Bump extensions/MirahezeMagic from `972c0f2` to `5cf7a35` - 13https://github.com/miraheze/mediawiki/pull/12680 [03:35:21] [02mediawiki] 07dependabot[bot] edited pull request 03#12680: Bump extensions/MirahezeMagic from `972c0f2` to `5cf7a35` - 13https://github.com/miraheze/mediawiki/pull/12680 [03:35:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_39/extensions/MirahezeMagic-6bc9b3b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b89da86cec58 [03:35:39] [02miraheze/mediawiki] 07dependabot[bot] 03b89da86 - Bump extensions/MirahezeMagic from `972c0f2` to `6bc9b3b` [03:35:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_39/extensions/MirahezeMagic-6bc9b3b - 13https://github.com/miraheze/mediawiki [03:35:42] [02mediawiki] 07dependabot[bot] opened pull request 03#12833: Bump extensions/MirahezeMagic from `972c0f2` to `6bc9b3b` - 13https://github.com/miraheze/mediawiki/pull/12833 [03:35:53] [02mediawiki] 07github-actions[bot] labeled pull request 03#12833: Bump extensions/MirahezeMagic from `972c0f2` to `6bc9b3b` - 13https://github.com/miraheze/mediawiki/pull/12833 [03:35:56] [02mediawiki] 07github-actions[bot] labeled pull request 03#12833: Bump extensions/MirahezeMagic from `972c0f2` to `6bc9b3b` - 13https://github.com/miraheze/mediawiki/pull/12833 [03:35:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#12833: Bump extensions/MirahezeMagic from `972c0f2` to `6bc9b3b` - 13https://github.com/miraheze/mediawiki/pull/12833 [03:39:19] [02mediawiki] 07Universal-Omega closed pull request 03#12833: Bump extensions/MirahezeMagic from `972c0f2` to `6bc9b3b` - 13https://github.com/miraheze/mediawiki/pull/12833 [03:39:22] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/d27583f91746...7aabceaadfc8 [03:39:25] [02miraheze/mediawiki] 07dependabot[bot] 037aabcea - Bump extensions/MirahezeMagic from `972c0f2` to `6bc9b3b` (#12833) [03:39:26] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_39/extensions/MirahezeMagic-6bc9b3b [03:39:27] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_39/extensions/MirahezeMagic-6bc9b3b - 13https://github.com/miraheze/mediawiki [03:39:28] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/7aabceaadfc8...76ec465a4715 [03:39:29] [02miraheze/mediawiki] 07Universal-Omega 0376ec465 - Update .branches.json [03:40:44] !log [@mwtask141] starting deploy of {'config': True} to all [03:40:50] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 5s [03:43:40] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/76ec465a4715...7191b9208164 [03:43:41] [02miraheze/mediawiki] 07Universal-Omega 037191b92 - Update Pivot [03:44:57] !log [universalomega@mwtask141] starting deploy of {'world': True} to all [03:50:22] [02MirahezeMagic] 07Universal-Omega deleted branch 03REL1_39 - 13https://github.com/miraheze/MirahezeMagic [03:50:23] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03REL1_39 [03:51:18] [02mw-config] 07Universal-Omega closed pull request 03#5085: Don't set DBO_DEFAULT flag - 13https://github.com/miraheze/mw-config/pull/5085 [03:51:20] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [03:51:21] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [03:54:45] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.72, 3.46, 3.07 [03:58:38] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.28, 3.29, 3.08 [04:01:27] !log [void@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=metawiki /srv/mediawiki/w/maintenance/archives/patch-templatelinks-tl_title-nullable.sql (END - exit=256) [04:10:17] !log [void@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=metawiki (END - exit=2) [04:10:38] !log [void@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=metawiki (START) [04:12:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.18, 4.41, 3.60 [04:14:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.82, 3.86, 3.48 [04:15:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/f0127cbee94d [04:15:22] [02miraheze/mw-config] 07Universal-Omega 03f0127cb - Remove PdfBook and Slack Notifications [04:15:24] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [04:15:25] [02mw-config] 07Universal-Omega opened pull request 03#5087: Remove PdfBook and Slack Notifications - 13https://github.com/miraheze/mw-config/pull/5087 [04:16:04] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f0127cbee94d...2dde66d9f722 [04:16:06] [02miraheze/mw-config] 07Universal-Omega 032dde66d - Update ManageWikiSettings.php [04:16:08] [02mw-config] 07Universal-Omega synchronize pull request 03#5087: Remove PdfBook and Slack Notifications - 13https://github.com/miraheze/mw-config/pull/5087 [04:16:25] miraheze/mw-config - Universal-Omega the build passed. [04:16:26] [discord] @CosmicAlpha can you also disable ReplaceText? [04:16:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2dde66d9f722...7322823d4f24 [04:16:47] [02miraheze/mw-config] 07Universal-Omega 037322823 - Update extension-list [04:16:50] [02mw-config] 07Universal-Omega synchronize pull request 03#5087: Remove PdfBook and Slack Notifications - 13https://github.com/miraheze/mw-config/pull/5087 [04:17:06] miraheze/mw-config - Universal-Omega the build passed. [04:17:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7322823d4f24...aa2089aec07a [04:17:44] [02miraheze/mw-config] 07Universal-Omega 03aa2089a - Update ManageWikiExtensions.php [04:17:47] [02mw-config] 07Universal-Omega synchronize pull request 03#5087: Remove PdfBook and Slack Notifications - 13https://github.com/miraheze/mw-config/pull/5087 [04:17:53] miraheze/mw-config - Universal-Omega the build passed. [04:18:17] [02mw-config] 07Universal-Omega edited pull request 03#5087: Remove PdfBook and Slack Notifications - 13https://github.com/miraheze/mw-config/pull/5087 [04:18:33] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.34, 3.10, 3.26 [04:18:42] miraheze/mw-config - Universal-Omega the build passed. [04:20:47] !log [universalomega@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 2150s [04:21:51] !log [universalomega@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/slacknotifications.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --disable (START) [04:22:02] !log [universalomega@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/slacknotifications.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --disable (END - exit=2) [04:22:17] !log [universalomega@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/slacknotifications.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --disable slacknotifications (START) [04:22:50] !log [universalomega@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/slacknotifications.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --disable slacknotifications (END - exit=65280) [04:23:23] !log [universalomega@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/pdfbook.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --disable pdfbook (START) [04:26:22] !log [universalomega@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/pdfbook.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --disable pdfbook (END - exit=0) [04:26:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.96, 4.45, 3.70 [04:28:42] !log sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/slacknotifications.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/populateWikiSettings.php --sourcelist=false --wgsetting=wgSlackIncomingWebhookUrl --remove [04:29:27] [02mw-config] 07Universal-Omega closed pull request 03#5087: Remove PdfBook and Slack Notifications - 13https://github.com/miraheze/mw-config/pull/5087 [04:29:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/mw-config/compare/8400652fd19f...3539c490ea5e [04:29:31] [02miraheze/mw-config] 07Universal-Omega 033539c49 - Remove PdfBook and Slack Notifications (#5087) [04:29:32] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [04:29:34] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [04:30:11] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [04:30:20] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [04:30:27] miraheze/mw-config - Universal-Omega the build passed. [04:30:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.23, 3.44, 3.49 [04:30:36] !log [universalomega@mwtask141] starting deploy of {'l10n': True} to all [04:32:33] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.19, 2.67, 3.20 [04:33:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3539c490ea5e...9bec5bc7ec3d [04:33:57] [02miraheze/mw-config] 07Universal-Omega 039bec5bc - Restrict ReplaceText [04:34:59] miraheze/mw-config - Universal-Omega the build passed. [04:35:29] [02miraheze/mw-config] 07Universal-Omega pushed 0331 commits to 03remove-ReplaceText [+0/-0/±55] 13https://github.com/miraheze/mw-config/compare/7225455ca9e7...8b2af533a95c [04:35:30] [02miraheze/mw-config] 07Universal-Omega 038b2af53 - Merge branch 'master' into remove-ReplaceText [04:35:32] [02mw-config] 07Universal-Omega synchronize pull request 03#5059: Remove ReplaceText - 13https://github.com/miraheze/mw-config/pull/5059 [04:36:31] miraheze/mw-config - Universal-Omega the build passed. [04:40:36] !log [@mwtask141] starting deploy of {'config': True} to all [04:40:53] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 17s [04:45:36] Maybe there shodl be a seperate feed channel [04:45:45] Kinda awkward to have one channel for both bot noise and for actual conversations [04:48:08] We were going to move some of the more noisy bots to #miraheze-sre-bots but like a lot of things these days, it hasn't gone anywhere. [04:48:55] botifico in particular. [04:50:47] ` like a lot of things these days, it hasn't gone anywhere.` [04:50:51] Can I say the R word? [04:52:42] !log [universalomega@mwtask141] finished deploy of {'l10n': True} to all - SUCCESS in 1325s [04:56:45] !log [@test131] starting deploy of {'config': True} to all [04:56:46] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [04:58:19] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:04:29] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.2165777981 secs [05:05:07] !log [universalomega@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/CheckUser/maintenance/populateCucActor.php (END - exit=0) [05:05:42] FINALLY! Done with everything! [05:24:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.65, 3.48, 2.65 [05:27:14] !log [@test131] starting deploy of {'config': True} to all [05:27:15] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [05:27:41] Everything for what? [05:27:50] Or just everything of everything? You paid the US national debt? [05:30:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.10, 3.00, 3.93 [05:30:29] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.01111242175 secs [05:30:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.47, 3.84, 3.19 [05:32:33] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.14, 2.90, 2.93 [05:32:44] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 3.36, 3.72, 3.99 [05:34:04] Naleksuh: lol, I meant everything for the 1.39 upgrade (SQL/migration scripts). [05:34:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 10.42, 5.62, 4.65 [05:34:44] PROBLEM - matomo131 Current Load on matomo131 is CRITICAL: CRITICAL - load average: 3.83, 3.86, 4.01 [05:36:07] I thought that was all done yesterday [05:36:25] Nope, scripts have been running almost non-stop since then. [05:36:44] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 2.92, 3.63, 3.91 [05:38:07] Why [05:38:26] Because they take a while to run on 6,500 wikis. [05:43:16] Yes, but if they're upgrade related, did the upgrade really complete before then? [05:47:00] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=arhetwiki /home/reception/linktarget.sql (END - exit=0) [05:48:44] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/archives/patch-templatelinks-target_id.sql --wiki=arhetwiki (END - exit=0) [05:50:36] Naleksuh: they weren't all super breaking issues, just needed to be done. [05:50:44] PROBLEM - matomo131 Current Load on matomo131 is CRITICAL: CRITICAL - load average: 4.89, 4.08, 3.91 [05:51:17] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=arhetwiki /srv/mediawiki/w/maintenance/archives/patch-templatelinks-target_id.sql (END - exit=0) [05:53:14] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=ezmccainwiki /srv/mediawiki/w/extensions/Moderation/sql/patch-moderation.sql (END - exit=0) [05:54:04] !log [void@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=houkai2ndwiki (START) [05:57:01] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 9.78, 4.90, 3.44 [05:57:22] !log [@test131] starting deploy of {'config': True} to all [05:57:23] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [05:58:44] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 3.05, 3.82, 3.90 [05:59:08] RECOVERY - mwtask141 Disk Space on mwtask141 is OK: DISK OK - free space: / 36103 MB (50% inode=91%); [06:01:45] !log [void@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=houkai2ndwiki (END - exit=0) [06:02:50] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.17, 3.04, 3.14 [06:03:42] !log [void@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=metawiki (END - exit=2) [06:04:07] !log [void@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/refreshLinks.php --wiki=metawiki (START) [06:08:44] PROBLEM - matomo131 Current Load on matomo131 is CRITICAL: CRITICAL - load average: 4.07, 3.70, 3.76 [06:10:44] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 3.25, 3.60, 3.72 [06:10:57] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.18, 3.51, 2.66 [06:12:03] !log [void@mon141] test [06:12:33] !log [void@mon141] test [06:12:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:13:57] !log [void@db101] MariaDB [metawiki]> CREATE TABLE tl2 LIKE templatelinks; [06:13:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:14:08] !log [void@db101] MariaDB [metawiki]> INSERT INTO tl2 SELECT * FROM templatelinks GROUP BY tl_from; [06:14:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:14:17] !log [void@db101] MariaDB [metawiki]> ALTER TABLE tl2 ADD PRIMARY KEY (tl_from, tl_target_id); [06:14:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:14:27] !log [void@db101] MariaDB [metawiki]> DROP TABLE templatelinks; ALTER TABLE tl2 RENAME TO templatelinks; [06:14:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:16:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.55, 3.82, 3.10 [06:18:44] PROBLEM - matomo131 Current Load on matomo131 is CRITICAL: CRITICAL - load average: 5.16, 4.01, 3.78 [06:20:44] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 3.99, 3.86, 3.75 [06:20:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.61, 3.35, 3.09 [06:22:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.31, 2.79, 3.96 [06:26:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.29, 3.93, 4.06 [06:26:44] PROBLEM - matomo131 Current Load on matomo131 is CRITICAL: CRITICAL - load average: 4.65, 4.06, 3.85 [06:27:03] !log [@test131] starting deploy of {'config': True} to all [06:27:04] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [06:27:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:27:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:28:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.40, 3.22, 3.78 [06:28:34] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.24, 4.09, 2.99 [06:30:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.19, 3.41, 2.88 [06:32:33] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.66, 2.76, 2.70 [06:34:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.63, 4.25, 3.93 [06:44:44] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 2.65, 3.77, 4.00 [06:48:44] PROBLEM - matomo131 Current Load on matomo131 is CRITICAL: CRITICAL - load average: 4.37, 3.93, 4.00 [06:50:44] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 3.99, 3.82, 3.94 [06:54:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.95, 2.68, 2.21 [06:56:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.66, 3.68, 2.63 [06:57:23] !log [@test131] starting deploy of {'config': True} to all [06:57:24] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [06:57:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:57:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:58:44] PROBLEM - matomo131 Current Load on matomo131 is CRITICAL: CRITICAL - load average: 4.01, 3.83, 3.87 [07:00:44] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 3.24, 3.77, 3.86 [07:04:44] PROBLEM - matomo131 Current Load on matomo131 is CRITICAL: CRITICAL - load average: 4.20, 3.80, 3.83 [07:06:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.34, 3.88, 3.96 [07:06:44] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 3.32, 3.65, 3.78 [07:10:33] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.74, 2.41, 3.31 [07:12:44] PROBLEM - matomo131 Current Load on matomo131 is CRITICAL: CRITICAL - load average: 5.09, 3.97, 3.82 [07:14:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.32, 2.54, 3.82 [07:14:44] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 3.50, 3.72, 3.74 [07:18:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.58, 2.02, 3.30 [07:24:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.23, 3.31, 2.91 [07:26:33] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.12, 3.33, 2.98 [07:27:03] !log [@test131] starting deploy of {'config': True} to all [07:27:04] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [07:27:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:27:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:28:44] RECOVERY - matomo131 Current Load on matomo131 is OK: OK - load average: 2.52, 2.97, 3.32 [07:36:44] PROBLEM - matomo131 Current Load on matomo131 is CRITICAL: CRITICAL - load average: 4.19, 3.67, 3.47 [07:37:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.69, 5.41, 3.78 [07:38:44] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 3.58, 3.57, 3.46 [07:42:44] RECOVERY - matomo131 Current Load on matomo131 is OK: OK - load average: 2.92, 3.23, 3.34 [07:43:08] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.18, 3.10, 3.34 [07:57:19] !log [@test131] starting deploy of {'config': True} to all [07:57:20] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [07:57:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:57:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:58:44] PROBLEM - matomo131 Current Load on matomo131 is CRITICAL: CRITICAL - load average: 4.05, 3.44, 3.24 [08:00:45] RECOVERY - matomo131 Current Load on matomo131 is OK: OK - load average: 2.79, 3.23, 3.20 [08:02:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.76, 3.50, 2.90 [08:04:30] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.84, 3.54, 2.98 [08:10:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.09, 3.08, 3.03 [08:11:31] PROBLEM - matomo131 Current Load on matomo131 is CRITICAL: CRITICAL - load average: 4.75, 3.74, 3.42 [08:23:08] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 2.48, 3.65, 3.65 [08:27:20] !log [@test131] starting deploy of {'config': True} to all [08:27:21] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [08:27:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:27:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:28:30] PROBLEM - vedopedia.witches-empire.com - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.405604839324951 seconds [08:30:53] RECOVERY - matomo131 Current Load on matomo131 is OK: OK - load average: 1.57, 2.80, 3.34 [08:34:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.21, 3.83, 2.92 [08:36:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.87, 3.62, 2.96 [08:38:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.43, 3.85, 3.11 [08:39:39] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 3.91, 3.43, 3.37 [08:42:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.48, 3.49, 3.18 [08:44:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.29, 2.70, 2.93 [08:45:28] RECOVERY - matomo131 Current Load on matomo131 is OK: OK - load average: 2.81, 3.29, 3.36 [08:49:22] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 3.17, 3.44, 3.42 [08:50:27] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=ttafwikiwiki --summary=Imported from https://talkingfriends.fandom.com --u=Miraheze maintenance script --search-recursively /home/reception/talkingfriendfandom/talkingfriendsfandomcom-20220626-wikidump/images (START) [08:50:28] [url] Talking Tom & Friends Wiki | Fandom | talkingfriends.fandom.com [08:50:28] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=ttafwikiwiki --summary=Imported from https://talkingfriends.fandom.com --u=Miraheze maintenance script --search-recursively /home/reception/talkingfriendfandom/talkingfriendsfandomcom-20220626-wikidump/images (END - exit=256) [08:50:28] [url] Talking Tom & Friends Wiki | Fandom | talkingfriends.fandom.com [08:50:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:50:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:50:51] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=ttafwikiwiki --summary=Imported from https://talkingfriends.fandom.com --u=Miraheze maintenance script /home/reception/talkingfriendfandom/talkingfriendsfandomcom-20220626-wikidump/images --search-recursively (START) [08:50:51] [url] Talking Tom & Friends Wiki | Fandom | talkingfriends.fandom.com [08:50:53] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=ttafwikiwiki --summary=Imported from https://talkingfriends.fandom.com --u=Miraheze maintenance script /home/reception/talkingfriendfandom/talkingfriendsfandomcom-20220626-wikidump/images --search-recursively (END - exit=256) [08:50:53] [url] Talking Tom & Friends Wiki | Fandom | talkingfriends.fandom.com [08:50:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:50:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:51:18] PROBLEM - matomo131 Current Load on matomo131 is CRITICAL: CRITICAL - load average: 4.34, 3.74, 3.53 [08:52:04] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=ttafwikiwiki --summary=Imported from https://talkingfriends.fandom.com --user=Miraheze maintenance script --search-recursively /home/reception/talkingfriendfandom/talkingfriendsfandomcom-20220626-wikidump/images (START) [08:52:05] [url] Talking Tom & Friends Wiki | Fandom | talkingfriends.fandom.com [08:52:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:53:15] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 3.74, 3.66, 3.52 [08:53:25] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [08:55:11] PROBLEM - matomo131 Current Load on matomo131 is CRITICAL: CRITICAL - load average: 4.47, 3.88, 3.61 [08:56:35] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.07, 3.27, 2.81 [08:56:35] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=ttafwikiwiki --summary=Imported from https://talkingfriends.fandom.com --user=Miraheze maintenance script --search-recursively /home/reception/talkingfriendfandom/talkingfriendsfandomcom-20220626-wikidump/images (END - exit=256) [08:56:36] [url] Talking Tom & Friends Wiki | Fandom | talkingfriends.fandom.com [08:56:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:57:01] !log [@test131] starting deploy of {'config': True} to all [08:57:02] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [08:57:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:57:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:57:07] PROBLEM - matomo131 Current Load on matomo131 is WARNING: WARNING - load average: 3.53, 3.78, 3.61 [08:58:32] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.05, 2.86, 2.72 [08:58:47] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=ttafwikiwiki --summary=Imported from https://talkingfriends.fandom.com --user=Miraheze maintenance script --search-recursively --memory-limit=1600000 /home/reception/talkingfriendfandom/talkingfriendsfandomcom-20220626-wikidump/images (START) [08:58:48] [url] Talking Tom & Friends Wiki | Fandom | talkingfriends.fandom.com [08:58:48] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=ttafwikiwiki --summary=Imported from https://talkingfriends.fandom.com --user=Miraheze maintenance script --search-recursively --memory-limit=1600000 /home/reception/talkingfriendfandom/talkingfriendsfandomcom-20220626-wikidump/images (END - exit=65280) [08:58:49] ... [08:58:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:58:54]