[00:00:15] !log [universalomega@test3] finished deploy of {'world': True} to skip [00:00:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:02:20] !log [universalomega@test3] starting deploy of {'world': True} to skip [00:02:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:02:38] !log [universalomega@test3] finished deploy of {'world': True} to skip [00:02:42] RECOVERY - cp12 Disk Space on cp12 is OK: DISK OK - free space: / 8309 MB (21% inode=96%); [00:02:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:02:55] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.08, 22.89, 19.22 [00:03:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.68, 7.42, 7.18 [00:04:12] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JBXZs [00:04:14] [02miraheze/MirahezeMagic] 07Universal-Omega 0351e710c - Update rebuildVersionCache.php [00:04:15] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#273: RebuildVersionCache: Support rebuilding for all extensions - 13https://git.io/JBPyf [00:04:30] !log [universalomega@test3] starting deploy of {'world': True} to skip [00:04:49] !log [universalomega@test3] finished deploy of {'world': True} to skip [00:04:55] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.42, 22.36, 19.47 [00:05:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:05:11] miraheze/MirahezeMagic - Universal-Omega the build passed. [00:05:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:06:06] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.43, 10.39, 8.25 [00:06:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.48, 24.68, 20.63 [00:07:56] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [00:08:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.03, 22.53, 20.30 [00:12:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.97, 7.98, 7.89 [00:12:52] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.42, 20.10, 19.82 [00:13:51] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JBXnF [00:13:52] [02miraheze/MirahezeMagic] 07Universal-Omega 03627532e - Update rebuildVersionCache.php [00:13:54] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#273: RebuildVersionCache: Support rebuilding for all extensions - 13https://git.io/JBPyf [00:14:09] !log [universalomega@test3] starting deploy of {'world': True} to skip [00:14:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:14:27] !log [universalomega@test3] finished deploy of {'world': True} to skip [00:14:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:14:50] miraheze/MirahezeMagic - Universal-Omega the build passed. [00:16:03] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.38, 8.30, 8.02 [00:17:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.49, 7.41, 7.81 [00:20:54] PROBLEM - bacula2 APT on bacula2 is CRITICAL: APT CRITICAL: 18 packages available for upgrade (5 critical updates). [00:26:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.67, 7.77, 7.95 [00:29:21] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JBXWX [00:29:23] [02miraheze/MirahezeMagic] 07Universal-Omega 03d5aebdb - Update rebuildVersionCache.php [00:29:24] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#273: RebuildVersionCache: Support rebuilding for all extensions - 13https://git.io/JBPyf [00:29:49] !log [universalomega@test3] starting deploy of {'world': True} to skip [00:29:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:30:05] !log [universalomega@test3] finished deploy of {'world': True} to skip [00:30:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:30:17] miraheze/MirahezeMagic - Universal-Omega the build passed. [00:32:26] !log [universalomega@test3] starting deploy of {'world': True} to skip [00:32:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:33:00] !log [universalomega@test3] finished deploy of {'world': True} to skip [00:33:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:33:22] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.82, 7.84, 7.64 [00:34:05] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.25, 8.61, 8.12 [00:35:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.25, 7.78, 7.66 [00:38:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.95, 7.71, 7.88 [00:43:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.04, 7.36, 7.32 [00:44:35] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#273: RebuildVersionCache: Support rebuilding for all extensions - 13https://git.io/JBPyf [00:44:37] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JBX4V [00:44:39] [02miraheze/MirahezeMagic] 07Universal-Omega 038cd278d - Update rebuildVersionCache.php [00:45:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.92, 7.32, 7.30 [00:45:36] miraheze/MirahezeMagic - Universal-Omega the build passed. [00:46:45] !log [universalomega@test3] starting deploy of {'world': True} to skip [00:47:00] !log [universalomega@test3] finished deploy of {'world': True} to skip [00:47:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:47:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:47:56] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 11 minutes ago with 0 failures [00:49:47] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JBXBM [00:49:49] [02miraheze/MirahezeMagic] 07Universal-Omega 03cabfd3d - Update rebuildVersionCache.php [00:49:50] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#273: RebuildVersionCache: Support rebuilding for all extensions - 13https://git.io/JBPyf [00:50:13] !log [universalomega@test3] starting deploy of {'world': True} to skip [00:50:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:50:32] !log [universalomega@test3] finished deploy of {'world': True} to skip [00:50:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:50:46] miraheze/MirahezeMagic - Universal-Omega the build passed. [00:54:15] [02MirahezeMagic] 07Universal-Omega edited pull request 03#273: RebuildVersionCache: Support rebuilding for all extensions - 13https://git.io/JBPyf [00:57:13] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.47, 5.80, 6.58 [01:00:04] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.63, 6.07, 6.70 [01:03:41] RECOVERY - bacula2 Bacula Private Git on bacula2 is OK: OK: Full, 5978 files, 32.72MB, 2021-08-01 01:03:00 (40.0 seconds ago) [01:06:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.09, 6.87, 6.86 [01:08:03] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.62, 6.41, 6.69 [01:36:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.38, 7.62, 6.83 [01:36:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.66, 6.92, 6.67 [01:37:57] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.74, 6.63, 6.56 [01:38:33] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.40, 6.27, 6.45 [01:51:10] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.83, 7.65, 6.82 [01:52:24] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.96, 8.24, 7.04 [01:53:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.33, 7.33, 6.80 [01:54:18] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.43, 7.52, 6.92 [01:57:00] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.96, 6.21, 6.48 [02:01:58] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.98, 6.49, 6.72 [02:11:30] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.29, 6.86, 6.57 [02:13:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.86, 6.35, 6.41 [02:17:18] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.79, 6.92, 6.60 [02:22:06] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.43, 7.09, 6.68 [02:25:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.90, 6.62, 6.64 [02:27:50] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.60, 7.53, 6.99 [02:29:46] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.43, 7.47, 7.03 [02:33:40] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.95, 8.32, 7.39 [02:35:51] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.75, 8.16, 7.26 [02:37:30] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.32, 7.74, 7.40 [02:37:48] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.29, 7.42, 7.09 [02:47:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.96, 6.35, 6.73 [02:49:13] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.43, 6.33, 6.78 [02:51:25] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.88, 6.87, 6.88 [02:53:22] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.94, 6.45, 6.72 [03:05:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.10, 9.14, 7.49 [03:06:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.14, 7.24, 6.83 [03:11:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.59, 7.31, 7.22 [03:14:03] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.65, 6.76, 6.75 [03:15:43] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JB1Tz [03:15:45] [02miraheze/MirahezeMagic] 07Universal-Omega 03dbaccc9 - Update rebuildVersionCache.php [03:15:46] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#273: RebuildVersionCache: Support rebuilding for all extensions - 13https://git.io/JBPyf [03:16:37] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JB1T5 [03:16:39] [02miraheze/MirahezeMagic] 07Universal-Omega 03af629fc - Update rebuildVersionCache.php [03:16:40] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#273: RebuildVersionCache: Support rebuilding for all extensions - 13https://git.io/JBPyf [03:16:54] miraheze/MirahezeMagic - Universal-Omega the build passed. [03:17:09] [02MirahezeMagic] 07Universal-Omega closed pull request 03#273: RebuildVersionCache: Support rebuilding for all extensions - 13https://git.io/JBPyf [03:17:10] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JB1Tx [03:17:12] [02miraheze/MirahezeMagic] 07Universal-Omega 035c927e7 - RebuildVersionCache: Support rebuilding for all extensions (#273) [03:17:13] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [03:17:15] [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/fQRGX [03:17:36] miraheze/MirahezeMagic - Universal-Omega the build passed. [03:18:03] miraheze/MirahezeMagic - Universal-Omega the build passed. [03:19:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.00, 8.58, 7.60 [03:22:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.53, 6.83, 6.78 [03:25:16] !log [universalomega@test3] starting deploy of {'world': True} to skip [03:25:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:25:37] !log [universalomega@test3] finished deploy of {'world': True} to skip [03:25:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:25:56] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:28:03] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.11, 7.88, 7.25 [03:29:16] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.44, 7.35, 7.60 [03:30:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.32, 7.48, 7.19 [03:32:03] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.42, 8.13, 7.46 [03:34:20] [02mediawiki] 07Universal-Omega closed pull request 03#2540: Bump extensions/Babel from `790c6c3` to `cfcc747` - 13https://git.io/J4asc [03:34:22] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JB1Yf [03:34:23] [02miraheze/mediawiki] 07dependabot[bot] 039aa983c - Bump extensions/Babel from `790c6c3` to `cfcc747` (#2540) [03:34:25] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/Babel-cfcc747 [03:34:26] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/Babel-cfcc747 - 13https://git.io/vbL5b [03:34:31] [02mediawiki] 07Universal-Omega closed pull request 03#2539: Bump extensions/Scribunto from `4044520` to `ac71012` - 13https://git.io/J4ass [03:34:33] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JB1Yt [03:34:34] [02miraheze/mediawiki] 07dependabot[bot] 037ffaf06 - Bump extensions/Scribunto from `4044520` to `ac71012` (#2539) [03:34:36] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/Scribunto-ac71012 [03:34:37] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_36/extensions/Scribunto-ac71012 - 13https://git.io/vbL5b [03:34:47] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.61, 19.16, 17.25 [03:36:50] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.93, 21.59, 18.34 [03:38:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.19, 21.48, 18.72 [03:39:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.02, 7.69, 7.66 [03:42:13] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [03:42:52] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.64, 19.40, 18.59 [03:43:14] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.58, 7.84, 7.78 [03:43:31] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [03:51:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.30, 7.51, 7.54 [03:53:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.35, 7.07, 7.38 [03:54:55] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.11, 7.13, 6.63 [03:56:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.04, 7.23, 7.93 [03:57:14] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.18, 7.63, 7.50 [03:58:04] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.65, 8.14, 8.17 [03:58:57] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.48, 6.73, 6.57 [04:02:57] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.43, 7.49, 6.35 [04:02:58] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.99, 8.00, 7.10 [04:04:57] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.78, 7.19, 6.36 [04:05:46] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [04:05:55] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.36, 22.24, 19.93 [04:06:13] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [04:07:55] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.54, 19.97, 19.38 [04:08:55] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.40, 7.96, 7.43 [04:11:19] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [04:12:36] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [04:12:55] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.50, 6.68, 6.54 [04:20:55] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.05, 5.93, 6.60 [04:25:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.79, 7.38, 7.89 [04:26:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.87, 7.45, 7.93 [04:28:03] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.73, 7.85, 8.01 [04:28:55] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.82, 6.68, 6.55 [04:30:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.30, 7.17, 7.74 [04:33:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.80, 8.60, 8.09 [04:34:03] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.26, 7.49, 7.65 [04:34:33] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [04:34:55] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.18, 6.75, 6.62 [04:35:18] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:36:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.50, 7.44, 7.62 [04:39:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.38, 7.54, 7.94 [04:39:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.12, 2.77 [04:43:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.23, 3.03, 2.80 [04:45:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.47, 7.83, 7.88 [04:48:03] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.10, 7.28, 7.16 [04:54:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.75, 7.43, 7.35 [04:58:17] PROBLEM - phabdigests.bots.miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate '*.bots.miraheze.wiki' expires in 15 day(s) (Tue 17 Aug 2021 04:56:02 GMT +0000). [04:59:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.81, 7.42, 7.91 [04:59:30] PROBLEM - phab.bots.miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate '*.bots.miraheze.wiki' expires in 15 day(s) (Tue 17 Aug 2021 04:56:02 GMT +0000). [04:59:52] PROBLEM - sopel.bots.miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate '*.bots.miraheze.wiki' expires in 15 day(s) (Tue 17 Aug 2021 04:56:02 GMT +0000). [05:02:09] PROBLEM - wildcard.bots.miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate '*.bots.miraheze.wiki' expires in 15 day(s) (Tue 17 Aug 2021 04:56:02 GMT +0000). [05:02:21] PROBLEM - icinga.bots.miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate '*.bots.miraheze.wiki' expires in 15 day(s) (Tue 17 Aug 2021 04:56:02 GMT +0000). [05:02:51] PROBLEM - phab-storage.bots.miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate '*.bots.miraheze.wiki' expires in 15 day(s) (Tue 17 Aug 2021 04:56:02 GMT +0000). [05:03:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.51, 7.67, 7.87 [05:04:05] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.95, 9.30, 8.09 [05:05:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.84, 7.83, 7.92 [05:07:14] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.75, 8.43, 8.12 [05:10:35] PROBLEM - wiki.thedev.gq - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.thedev.gq' expires in 15 day(s) (Tue 17 Aug 2021 05:08:15 GMT +0000). [05:12:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.02, 7.69, 7.77 [05:13:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.72, 7.72, 7.90 [05:15:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.28, 7.90, 7.94 [05:16:20] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JB1XF [05:16:22] [02miraheze/ssl] 07MirahezeSSLBot 030892798 - Bot: Update SSL cert for wiki.thedev.gq [05:18:03] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.38, 8.13, 7.89 [05:22:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.79, 7.86, 7.87 [05:25:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.06, 7.87, 7.95 [05:30:54] RECOVERY - wiki.thedev.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thedev.gq' will expire on Sat 30 Oct 2021 04:16:12 GMT +0000. [05:33:16] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.00, 8.05, 7.85 [05:34:04] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.82, 8.79, 8.10 [05:38:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.82, 7.92, 7.92 [05:40:05] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.10, 8.08, 7.98 [05:43:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.96, 7.49, 7.87 [05:46:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.96, 7.86, 7.93 [05:48:03] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.36, 8.00, 7.97 [05:50:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.83, 7.33, 7.74 [05:55:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.78, 7.83, 7.57 [05:57:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.75, 7.76, 7.58 [06:03:14] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.05, 7.38, 7.40 [06:04:05] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.12, 7.08, 7.18 [06:09:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.07, 7.66, 7.59 [06:10:08] [02mediawiki] 07Reception123 closed pull request 03#2289: T6471: GitInfo hack - 13https://git.io/Jn6nf [06:10:10] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JB1Ns [06:10:11] [02miraheze/mediawiki] 07Universal-Omega 033ba7eed - T6471: GitInfo hack (#2289) [06:11:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.86, 3.28, 2.78 [06:13:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 3.09, 2.78 [06:13:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.43, 8.72, 8.02 [06:13:53] [02puppet] 07Reception123 closed pull request 03#1838: mediawiki: automatically sync config - 13https://git.io/JBPY9 [06:13:55] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JB1AR [06:13:57] [02miraheze/puppet] 07RhinosF1 030651ebd - mediawiki: automatically sync config (#1838) [06:16:48] [02puppet] 07Reception123 commented on pull request 03#1789: Update default.vcl - 13https://git.io/JB1xE [06:20:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JB1p4 [06:20:13] [02miraheze/services] 07MirahezeSSLBot 030759648 - BOT: Updating services config for wikis [06:21:06] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JB1pD [06:21:08] [02miraheze/mediawiki] 07Reception123 0357ab3c5 - Update MirahezeMagic [06:26:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.61, 7.49, 7.89 [06:29:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.49, 7.47, 7.96 [06:33:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.31, 7.57, 7.86 [06:34:04] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.40, 9.35, 8.33 [06:35:56] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:36:59] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.86, 6.53, 5.90 [06:37:01] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.70, 19.18, 17.31 [06:37:48] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.77, 6.90, 6.09 [06:38:58] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.07, 7.33, 6.29 [06:39:43] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.24, 6.68, 6.11 [06:40:55] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.47, 6.55, 6.13 [06:41:02] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.27, 19.41, 17.87 [06:42:15] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:42:45] [02puppet] 07Reception123 closed pull request 03#1834: T7718: Remove stunnel check for test3 - 13https://git.io/JBVNs [06:42:46] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JBMfp [06:42:48] [02miraheze/puppet] 07Universal-Omega 03e595318 - T7718: Remove stunnel check for test3 (#1834) [06:43:39] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [06:54:49] PROBLEM - cp12 Stunnel Http for test3 on cp12 is UNKNOWN: NRPE: Command 'check_stunnel_test3' not defined [06:56:52] RECOVERY - bacula2 APT on bacula2 is OK: APT OK: 13 packages available for upgrade (0 critical updates). [06:57:35] Reception123: ^ not sure why it would, as it is nowhere in puppet now, but maybe just because puppet hasn't run on all servers yet but I'm not sure. [07:01:37] PROBLEM - cp13 Stunnel Http for test3 on cp13 is UNKNOWN: NRPE: Command 'check_stunnel_test3' not defined [07:05:36] yeah, I checked too to make sure and I didn't see it anywhere else [07:05:49] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:06:12] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:12:37] PROBLEM - cp3 Stunnel Http for test3 on cp3 is UNKNOWN: NRPE: Command 'check_stunnel_test3' not defined [07:20:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.02, 6.36, 7.99 [07:21:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.91, 6.54, 8.00 [07:23:34] PROBLEM - arru.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.arru.xyz' expires in 15 day(s) (Tue 17 Aug 2021 07:21:27 GMT +0000). [07:23:58] PROBLEM - www.arru.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.arru.xyz' expires in 15 day(s) (Tue 17 Aug 2021 07:21:27 GMT +0000). [07:27:07] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.02, 7.02, 7.71 [07:29:03] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.59, 6.51, 7.45 [07:30:10] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBMZc [07:30:11] [02miraheze/ssl] 07MirahezeSSLBot 03120c8c1 - Bot: Update SSL cert for www.arru.xyz [07:32:56] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.96, 7.58, 7.69 [07:33:44] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.95, 7.79, 7.53 [07:33:47] [02puppet] 07RhinosF1 commented on pull request 03#1835: hieradata: set values for sync scripts - 13https://git.io/JBMn4 [07:35:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.43, 7.90, 7.63 [07:36:48] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.87, 7.47, 7.66 [07:39:05] Reception123: can you merge the puppet pr too [07:39:21] Otherwise that most definately won't work [07:39:30] The set values one [07:40:12] you said it would break mw10 though [07:41:56] Reception123: yeah it won't [07:42:14] I realised I'd done it so it wouldn't [07:42:24] ok. I'll keep an eye on it just to be sure then [07:42:37] [02puppet] 07Reception123 closed pull request 03#1835: hieradata: set values for sync scripts - 13https://git.io/JBVNb [07:42:39] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JBMW3 [07:42:40] [02miraheze/puppet] 07RhinosF1 039e01064 - hieradata: set values for sync scripts (#1835) [07:44:33] Universal_Omega: did it work well last night? [07:46:10] RhinosF1: seemed to. I got my gitinfo fixes done now. Although the new stuff makes things fairly simple, but I do wish there was a way to make it not log every time, especially on test3. Would that be possible maybe? [07:46:38] Universal_Omega: I mean yeah it's very possible [07:46:41] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.74, 7.95, 7.70 [07:47:28] RhinosF1: yeah I just think because of the amount of test and deploys done sometimes on test3 all that logging is kind of excessive. [07:47:59] Yeah, I'll add a --no-log parameter [07:48:42] I seen how busy SAL is [07:49:05] Thanks! Other then that it makes things so much easier to do everything like i18n, etc... in one command. I really do like the change. [07:49:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.87, 8.19, 7.60 [07:49:22] Either that or I might move the log [07:49:31] To a seperate sync log [07:50:19] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Function lookup() did not find a value for the name 'mediawiki::default_sync' on node test3.miraheze.org [07:50:31] Reception123: after merging both? [07:50:46] RhinosF1: I think a `--no-log` parameter is sufficient enough but up to you. [07:51:02] after the second it seems to have changed to this: [07:51:03] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not find resource 'Git::Clone[MediaWiki Config]' in parameter 'subscribe' (file: /etc/puppetlabs/puppet/environments/production/modules/mediawiki/manifests/init.pp, line: 52) on node test3.miraheze.org [07:51:17] Reception123: that's the better one [07:51:38] Maybe that should be miraheze/mw-config [07:54:25] you mean like File[/srv/mediawiki/config] ? [07:55:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.89, 7.85, 7.71 [07:56:22] Reception123: no on the line that says subscribe [07:58:30] yeah but what exactly do you want to replace that with? [07:58:36] RECOVERY - www.arru.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'www.arru.xyz' will expire on Sat 30 Oct 2021 06:30:00 GMT +0000. [07:58:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.78, 7.55, 7.95 [07:58:57] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.74, 7.76, 7.65 [08:00:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.71, 8.62, 8.31 [08:02:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.43, 7.59, 7.96 [08:02:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.85, 7.81, 7.74 [08:04:15] RECOVERY - arru.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'www.arru.xyz' will expire on Sat 30 Oct 2021 06:30:00 GMT +0000. [08:04:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.47, 8.83, 8.39 [08:04:46] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.62, 8.88, 8.14 [08:06:42] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.95, 7.70, 7.79 [08:10:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.09, 7.35, 7.87 [08:11:56] !log [reception@jobrunner3] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=twistedwonderlandwiki /home/reception/twistedwonderland_pages_full.xml --username-prefix wikia:twisted-wonderland [08:12:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:14:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.45, 7.52, 7.61 [08:16:12] [02puppet] 07RhinosF1 opened pull request 03#1839: Update init.pp - 13https://git.io/JBM2z [08:16:29] Reception123: ^ [08:18:00] [02puppet] 07Reception123 closed pull request 03#1839: Update init.pp - 13https://git.io/JBM2z [08:18:02] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBMaY [08:18:03] [02miraheze/puppet] 07RhinosF1 0308e09d9 - Update init.pp (#1839) [08:18:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.68, 7.14, 7.44 [08:19:32] doesn't like that either [08:19:33] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not find resource 'Git::Clone[miraheze/mw-config]' in parameter 'subscribe' (file: /etc/puppetlabs/puppet/environments/production/modules/mediawiki/manifests/init.pp, line: 52) on node test3.miraheze.org [08:22:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.35, 7.49, 7.50 [08:24:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.35, 6.99, 7.20 [08:25:38] Reception123: what the hell is the resource called then [08:25:40] Hang on [08:26:20] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBMwN [08:26:22] [02miraheze/mw-config] 07RhinosF1 030323114 - Update README.md [08:26:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.61, 7.54, 7.54 [08:26:31] Reception123: run puppet and message me the output [08:26:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.73, 6.55, 7.02 [08:27:15] miraheze/mw-config - RhinosF1 the build passed. [08:27:55] not sure what you thought that would do but same output [08:28:32] Reception123: given I just updated config, a line about it [08:28:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.01, 7.00, 7.12 [08:28:43] well there won't be one if puppet fails before [08:28:58] Reception123: you'll have to revert my 2 changes [08:29:07] No [08:29:08] No [08:29:14] Try another mw* [08:29:18] ah, maybe you wanted this from jobrunner [08:29:18] Notice: /Stage[main]/Mediawiki/Git::Clone[MediaWiki config]/Exec[git_pull_MediaWiki config]/returns: executed successfully (corrective) [08:29:27] Yes [08:29:34] But that's what I did the first time [08:29:36] I think I see it [08:29:39] you did [08:29:40] subscribe => Git::Clone['MediaWiki Config'], [08:29:43] but it's config lowercase [08:29:48] Ye! [08:29:57] let's give it a try [08:30:06] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBMoC [08:30:07] [02miraheze/puppet] 07Reception123 03284f40f - fix [08:30:36] [02puppet] 07RhinosF1 opened pull request 03#1840: Urgh - 13https://git.io/JBMo6 [08:30:51] [02puppet] 07RhinosF1 closed pull request 03#1840: Urgh - 13https://git.io/JBMo6 [08:31:19] now it's mad about something else [08:31:20] Error: Failed to apply catalog: Validation of Exec[MediaWiki Config Sync] failed: 'deploy-mediawiki' is not qualified and no path was specified. Please qualify the command or specify a path. (file: /etc/puppetlabs/puppet/environments/production/modules/mediawiki/manifests/init.pp, line: 47) [08:32:33] That's easy [08:33:07] yup, I'll let you do the PR :) [08:34:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.22, 8.49, 7.85 [08:36:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.85, 7.98, 7.76 [08:38:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.40, 8.15, 7.84 [08:38:41] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.90, 7.63, 7.55 [08:39:20] [02puppet] 07RhinosF1 opened pull request 03#1841: Update init.pp - 13https://git.io/JBMiV [08:39:29] Reception123: ^ [08:40:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.20, 7.61, 7.67 [08:40:49] [02puppet] 07Reception123 closed pull request 03#1841: Update init.pp - 13https://git.io/JBMiV [08:40:50] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBMid [08:40:52] [02miraheze/puppet] 07RhinosF1 037587098 - Update init.pp (#1841) [08:41:51] Error: /Stage[main]/Mediawiki/Exec[MediaWiki Config Sync]: Failed to call refresh: '/usr/local/bin/deploy-mediawiki --config --sync=skip' returned 2 instead of one of [0] [08:41:51] Error: /Stage[main]/Mediawiki/Exec[MediaWiki Config Sync]: '/usr/local/bin/deploy-mediawiki --config --sync=skip' returned 2 instead of one of [0] [08:44:32] Reception123: it must give an output [08:44:35] Of the script [08:44:47] Some indication of why it exited with code 2 [08:45:08] That's been ran countless times [08:45:14] https://www.irccloud.com/pastebin/mLXvYx1Z/ [08:46:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.08, 7.59, 7.61 [08:46:42] [02puppet] 07RhinosF1 opened pull request 03#1842: Update init.pp - 13https://git.io/JBMXw [08:46:49] Reception123: ^ [08:47:08] hopefully this is the last [08:47:15] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBMX9 [08:47:17] [02miraheze/puppet] 07RhinosF1 03ec6bca2 - Update init.pp (#1842) [08:47:18] [02puppet] 07Reception123 closed pull request 03#1842: Update init.pp - 13https://git.io/JBMXw [08:48:11] works now [08:48:50] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [08:49:08] Reception123: did it actually execute [08:49:26] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBM1R [08:49:28] [02miraheze/mw-config] 07RhinosF1 032ad67c6 - Update README.md [08:49:40] Reception123: I took off my . so let's run [08:50:08] just a regular run [08:50:10] https://www.irccloud.com/pastebin/iTo2XsJC/ [08:50:22] miraheze/mw-config - RhinosF1 the build passed. [08:50:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.96, 7.79, 7.72 [08:50:50] !log [@test3] starting deploy of {'config': True} to skip [08:50:51] !log [@test3] finished deploy of {'config': True} to skip [08:50:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:50:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:51:04] after the readme change: [08:51:06] https://www.irccloud.com/pastebin/8g1D6YrJ/ [08:51:22] Reception123: it worked [08:51:30] I know because we have a log entry [08:51:36] And it's subscribed perfectly [08:51:57] I will add a --no-log or move log to another log later to we don't fill SAL [08:52:17] Yay! [08:52:38] But \o/ [08:52:43] Config syncs now just happen [08:54:01] * RhinosF1 finds this system much better [08:54:10] Less funky commands to remember [08:54:42] I just need to get my head round ssh agent now [09:00:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.25, 5.96, 6.67 [09:04:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.24, 6.11, 6.70 [09:34:45] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.51, 7.56, 6.55 [09:36:41] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.93, 6.50, 6.28 [10:04:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 13.10, 8.48, 6.80 [10:05:36] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.39, 7.90, 6.71 [10:08:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.77, 6.94, 6.61 [10:10:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.83, 6.62, 6.53 [10:11:27] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.88, 6.37, 6.43 [10:25:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBDmR [10:25:13] [02miraheze/services] 07MirahezeSSLBot 034683405 - BOT: Updating services config for wikis [10:35:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.33, 6.77, 6.27 [10:37:19] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.47, 6.25, 6.14 [10:49:38] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.19, 6.71, 6.15 [10:51:36] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.27, 6.55, 6.14 [10:55:29] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.72, 6.80, 6.35 [11:05:19] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.31, 6.82, 6.49 [11:07:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.44, 6.33, 6.35 [11:35:29] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.62, 7.28, 6.41 [11:39:23] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.74, 6.53, 6.30 [11:54:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.88, 6.82, 6.54 [11:57:54] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.79, 6.59, 6.57 [12:03:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.20, 7.13, 6.64 [12:03:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.07, 7.70, 6.98 [12:26:33] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.89, 7.76, 7.84 [12:34:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.23, 8.55, 8.03 [12:36:29] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.62, 7.94, 7.86 [12:36:55] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.62, 7.41, 7.94 [12:56:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.11, 5.90, 6.64 [13:04:29] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.62, 7.47, 6.94 [13:04:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.09, 8.32, 7.52 [13:05:50] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.50, 21.19, 17.89 [13:06:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.85, 6.88, 6.79 [13:06:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.64, 7.77, 7.43 [13:07:47] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.31, 18.00, 17.12 [13:08:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.06, 6.34, 6.61 [13:14:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.86, 6.14, 6.72 [13:26:40] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.50, 6.81, 6.57 [13:27:36] PROBLEM - ns1 APT on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:28:24] ? [13:28:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.63, 6.25, 6.38 [13:29:02] Reception123: ns1 looks upset [13:29:32] RECOVERY - ns1 APT on ns1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:31:31] something isn't right on that [13:31:37] Load spiked super [13:31:48] Might be worth checking kernel logs [13:35:12] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 17.57, 10.71, 7.86 [13:36:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.41, 6.83, 6.58 [13:38:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.08, 6.25, 6.41 [13:41:02] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.26, 8.00, 7.64 [13:50:41] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.08, 7.04, 7.12 [13:56:31] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.73, 7.01, 7.13 [13:57:09] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.48, 19.90, 17.45 [13:57:35] PROBLEM - ns1 APT on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:59:06] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.94, 18.70, 17.29 [13:59:32] RECOVERY - ns1 APT on ns1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:00:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.71, 7.56, 7.28 [14:01:33] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.18, 6.80, 6.36 [14:06:56] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.06, 20.71, 18.62 [14:08:55] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.59, 19.26, 18.35 [14:09:12] PROBLEM - cp13 Current Load on cp13 is WARNING: WARNING - load average: 1.48, 1.71, 1.23 [14:11:12] RECOVERY - cp13 Current Load on cp13 is OK: OK - load average: 0.40, 1.23, 1.11 [14:11:14] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.64, 8.06, 7.25 [14:13:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.57, 7.60, 7.19 [14:15:39] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.03, 6.37, 5.73 [14:17:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.23, 7.93, 7.40 [14:17:38] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.53, 5.89, 5.63 [14:19:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.70, 7.60, 7.33 [14:19:13] 2021-08-01 13:57:53.000 +00:00 ns1.miraheze.org [14:19:14] OOM killed process 19263 (puppet) total-vm:232080kB, anon-rss:56540kB, file-rss:64kB [14:19:14] 2021-08-01 13:56:06.000 +00:00 ns1.miraheze.org [14:19:14] OOM killed process 19329 (apt-mark) total-vm:58588kB, anon-rss:12564kB, file-rss:20kB [14:19:15] RhinosF1 ^ [14:19:20] seems to be over now [14:20:58] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.37, 8.36, 7.64 [14:21:31] PROBLEM - www.mcpk.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mcpk.wiki could not be found [14:21:32] PROBLEM - wiki.rebirthofthenight.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.rebirthofthenight.com could not be found [14:21:33] PROBLEM - wiki.kourouklides.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kourouklides.com could not be found [14:21:33] PROBLEM - wiki.velaan.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.velaan.org could not be found [14:21:36] PROBLEM - rangpurpedia.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rangpurpedia.xyz could not be found [14:21:42] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.85, 20.88, 19.41 [14:21:54] PROBLEM - en.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.gyaanipedia.com could not be found [14:21:56] PROBLEM - thelonsdalebattalion.co.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for thelonsdalebattalion.co.uk could not be found [14:21:57] PROBLEM - wiki.joust.ro - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.joust.ro could not be found [14:21:58] PROBLEM - guia.cineastas.pt - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for guia.cineastas.pt could not be found [14:22:01] PROBLEM - cubashpedia.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cubashpedia.cf could not be found [14:22:02] PROBLEM - patternarchive.online - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for patternarchive.online could not be found [14:22:54] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.38, 7.60, 7.45 [14:23:40] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.19, 18.78, 18.85 [14:28:02] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet last ran 1 hour ago [14:28:23] RECOVERY - www.mcpk.wiki - reverse DNS on sslhost is OK: SSL OK - www.mcpk.wiki reverse DNS resolves to cp13.miraheze.org - CNAME OK [14:28:25] RECOVERY - wiki.kourouklides.com - reverse DNS on sslhost is OK: SSL OK - wiki.kourouklides.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [14:28:25] RECOVERY - wiki.rebirthofthenight.com - reverse DNS on sslhost is OK: SSL OK - wiki.rebirthofthenight.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [14:28:33] RECOVERY - wiki.velaan.org - reverse DNS on sslhost is OK: SSL OK - wiki.velaan.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [14:28:34] RECOVERY - rangpurpedia.xyz - reverse DNS on sslhost is OK: SSL OK - rangpurpedia.xyz reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [14:28:42] RECOVERY - cubashpedia.cf - reverse DNS on sslhost is OK: SSL OK - cubashpedia.cf reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [14:28:44] RECOVERY - patternarchive.online - reverse DNS on sslhost is OK: SSL OK - patternarchive.online reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [14:28:46] RECOVERY - en.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - en.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [14:28:52] RECOVERY - wiki.joust.ro - reverse DNS on sslhost is OK: SSL OK - wiki.joust.ro reverse DNS resolves to cp13.miraheze.org - CNAME OK [14:28:53] RECOVERY - thelonsdalebattalion.co.uk - reverse DNS on sslhost is OK: SSL OK - thelonsdalebattalion.co.uk reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [14:28:55] RECOVERY - guia.cineastas.pt - reverse DNS on sslhost is OK: SSL OK - guia.cineastas.pt reverse DNS resolves to cp13.miraheze.org - CNAME OK [14:30:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.67, 7.10, 7.72 [14:31:56] Reception123: OOM? Lovely [14:32:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.59, 7.27, 7.22 [14:34:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.68, 8.54, 8.12 [14:35:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.03, 2.72 [14:36:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.90, 7.76, 7.88 [14:36:41] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.99, 7.82, 7.55 [14:37:12] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.03, 3.00, 2.74 [14:39:09] [02mw-config] 07thelevking opened pull request 03#4032: Make AMC mode default for registered users on isvwiki - 13https://git.io/JByzk [14:40:01] miraheze/mw-config - thelevking the build passed. [14:40:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.48, 8.68, 8.19 [14:40:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.62, 8.18, 7.73 [14:41:19] I left a note there about wanting to add a new variable, if someone can explain to me how to do it safely (unless it is just "add a new array"), it would be awesome. [14:43:34] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.73, 7.30, 6.45 [14:45:32] Great job on the ManageWiki form update, btw. [14:49:35] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.43, 6.56, 6.43 [14:55:56] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:06:58] [[Extension:UserFunctions]] seems like some functions of this extension are not working (requiring some changes in LocalSettings.php, not available on ManageWiki) [15:06:59] https://meta.miraheze.org/wiki/Extension:UserFunctions?action=edit&redlink=1 [15:07:01] [url] Extension:UserFunctions - Miraheze Meta | meta.miraheze.org [15:07:41] oops [15:07:44] [[mw:Extension:UserFunctions]] [15:07:45] https://www.mediawiki.org/wiki/Extension:UserFunctions [15:07:45] [url] Extension:UserFunctions - MediaWiki | www.mediawiki.org [15:09:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.34, 2.87 [15:11:18] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.77, 3.23, 2.90 [15:11:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.51, 7.42, 6.84 [15:13:29] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.04, 8.14, 7.18 [15:15:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.58, 7.65, 7.12 [15:19:29] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.65, 6.26, 6.70 [15:23:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.90, 6.71, 6.80 [15:27:29] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.11, 6.64, 6.77 [15:35:55] solve this fast [15:35:55] https://phabricator.miraheze.org/T7730 [15:35:56] [url] ⚓ T7730 Comments disappear after editing (crappygames.miraheze.org) | phabricator.miraheze.org [15:45:25] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.39, 6.96, 6.73 [15:47:24] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.31, 7.83, 7.08 [15:49:24] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.28, 7.50, 7.08 [15:53:25] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.96, 6.45, 6.75 [15:54:19] @MarioMario456: please read [[CoC]] [15:54:28] We ask you be polite [15:58:20] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet last ran 1 hour ago [15:58:51] [02mw-config] 07Universal-Omega closed pull request 03#4032: Make AMC mode default for registered users on isvwiki - 13https://git.io/JByzk [15:58:52] Reception123: ^ [15:58:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JByd8 [15:58:54] [02miraheze/mw-config] 07thelevking 036fb16f3 - Make AMC mode default for users on isvwiki (#4032) [15:59:41] Reception123: mem usage seems to be high [15:59:43] Reboot? [15:59:46] miraheze/mw-config - Universal-Omega the build passed. [16:00:52] https://phabricator.miraheze.org/T7731 [16:00:53] [url] ⚓ T7731 ns1 mem usage jumped | phabricator.miraheze.org [16:01:43] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JByFW [16:01:44] [02miraheze/ManageWiki] 07Universal-Omega 03873466b - Update ManageWikiFormFactoryBuilder.php [16:01:46] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vpSns [16:02:04] [02ManageWiki] 07Universal-Omega opened pull request 03#296: Add help-message to 'description' field - 13https://git.io/JByFu [16:02:43] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JByFw [16:02:44] [02miraheze/ManageWiki] 07Universal-Omega 03d75a4ce - Update ManageWikiFormFactoryBuilder.php [16:02:46] [02ManageWiki] 07Universal-Omega synchronize pull request 03#296: Add help-message to 'description' field - 13https://git.io/JByFu [16:02:47] !log [@test3] starting deploy of {'config': True} to skip [16:02:48] !log [@test3] finished deploy of {'config': True} to skip [16:02:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:02:59] miraheze/ManageWiki - Universal-Omega the build passed. [16:02:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:03:38] miraheze/ManageWiki - Universal-Omega the build passed. [16:07:10] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JByFx [16:07:11] [02miraheze/ManageWiki] 07Universal-Omega 0314b06c8 - Update en.json [16:07:13] [02ManageWiki] 07Universal-Omega synchronize pull request 03#296: Add help-message to 'description' field - 13https://git.io/JByFu [16:08:08] miraheze/ManageWiki - Universal-Omega the build passed. [16:08:14] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.25, 7.23, 6.54 [16:08:23] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JBybq [16:08:25] [02miraheze/ManageWiki] 07Universal-Omega 031d27085 - Update qqq.json [16:08:26] [02ManageWiki] 07Universal-Omega synchronize pull request 03#296: Add help-message to 'description' field - 13https://git.io/JByFu [16:09:20] miraheze/ManageWiki - Universal-Omega the build passed. [16:10:10] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.85, 7.18, 6.61 [16:12:06] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.66, 7.67, 6.85 [16:13:24] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.92, 7.56, 6.94 [16:14:02] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.36, 7.44, 6.86 [16:16:26] RhinosF1: I'll check now [16:17:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 7.97, 8.02, 7.28 [16:19:24] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.91, 7.44, 7.17 [16:21:24] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.66, 8.34, 7.51 [16:21:36] RhinosF1: will have to reboot as puppet is getting killed [16:21:48] !log rebooted ns1 [16:21:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:23:23] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.26, 7.17, 7.19 [16:23:57] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:31:17] K [16:32:58] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JByjV [16:32:59] [02miraheze/MirahezeMagic] 07Universal-Omega 03d1c05f7 - Update MirahezeMagicHooks.php [16:33:01] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#266: Update ManageWiki database type when renaming or deleting wikis - 13https://git.io/JnKfd [16:33:50] miraheze/MirahezeMagic - Universal-Omega the build passed. [16:41:20] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.08, 7.50, 7.22 [16:43:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.53, 7.06, 7.09 [16:47:28] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.32, 6.61, 6.79 [16:54:56] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.23, 7.27, 7.05 [16:56:52] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.62, 6.94, 6.95 [16:57:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.84, 6.60, 6.63 [16:59:30] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.54, 6.57, 6.61 [17:03:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.12, 6.98, 6.77 [17:04:38] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.14, 7.55, 7.17 [17:05:30] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.26, 6.66, 6.68 [17:06:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.73, 7.35, 7.14 [17:09:30] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.12, 7.25, 6.87 [17:11:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.54, 7.28, 6.93 [17:14:35] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://git.io/JBStt [17:14:36] [02miraheze/DataDump] 07Universal-Omega 032353c85 - Create ApiGenerateDumps.php [17:14:38] [02DataDump] 07Universal-Omega synchronize pull request 03#32: T7341: Create API for deleting dumps - 13https://git.io/JBPaH [17:15:04] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBSt3 [17:15:05] [02miraheze/DataDump] 07Universal-Omega 03d76d4e6 - Update ApiDeleteDumps.php [17:15:07] [02DataDump] 07Universal-Omega synchronize pull request 03#32: T7341: Create API for deleting dumps - 13https://git.io/JBPaH [17:15:30] miraheze/DataDump - Universal-Omega the build passed. [17:15:57] miraheze/DataDump - Universal-Omega the build passed. [17:16:23] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBSt2 [17:16:25] [02miraheze/DataDump] 07Universal-Omega 037fad69a - Update extension.json [17:16:26] [02DataDump] 07Universal-Omega synchronize pull request 03#32: T7341: Create API for deleting dumps - 13https://git.io/JBPaH [17:16:44] [02DataDump] 07Universal-Omega edited pull request 03#32: T7341: Create API modules for generating and deleting dumps - 13https://git.io/JBPaH [17:17:19] miraheze/DataDump - Universal-Omega the build passed. [17:19:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.29, 7.60, 7.12 [17:20:20] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.53, 6.69, 6.80 [17:20:57] [02DataDump] 07Universal-Omega closed pull request 03#32: T7341: Create API modules for generating and deleting dumps - 13https://git.io/JBPaH [17:20:59] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03master [+2/-0/±1] 13https://git.io/JBSqH [17:21:00] [02miraheze/DataDump] 07Universal-Omega 03820feb6 - T7341: Create API modules for generating and deleting dumps (#32) [17:21:02] [02DataDump] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fhhKV [17:21:03] [02miraheze/DataDump] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [17:21:52] miraheze/DataDump - Universal-Omega the build passed. [17:27:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.54, 7.83, 7.58 [17:28:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.16, 6.92, 6.85 [17:29:44] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JBSOm [17:29:45] [02miraheze/MirahezeMagic] 07Universal-Omega 03dbe6171 - Update populateWikibaseSitesTable.php [17:29:47] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#274: T6045: Maintain default Wikimedia sites - 13https://git.io/JBXkb [17:30:42] miraheze/MirahezeMagic - Universal-Omega the build passed. [17:35:34] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.03, 7.46, 7.42 [17:38:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.88, 8.23, 7.46 [17:39:04] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://git.io/JBSsk [17:39:05] [02miraheze/MirahezeMagic] 07Universal-Omega 03d65b29e - Update populateWikibaseSitesTable.php [17:39:07] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#274: T6045: Maintain default Wikimedia sites - 13https://git.io/JBXkb [17:39:36] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.37, 7.91, 7.65 [17:40:02] miraheze/MirahezeMagic - Universal-Omega the build passed. [17:42:20] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.55, 7.43, 7.33 [17:48:20] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.26, 7.28, 7.25 [17:50:20] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.53, 6.76, 7.05 [17:53:32] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.52, 7.47, 7.42 [17:55:32] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.45, 7.00, 7.25 [17:56:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.13, 6.23, 6.72 [17:59:31] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.65, 7.66, 7.44 [18:00:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.85, 7.17, 7.03 [18:03:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.71, 7.75, 7.53 [18:05:30] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.84, 8.18, 7.72 [18:07:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.19, 7.45, 7.51 [18:09:29] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.35, 7.75, 7.61 [18:11:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.37, 7.63, 7.59 [18:17:28] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.57, 8.01, 7.68 [18:19:29] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JBSlg [18:19:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.03, 7.88, 7.69 [18:19:30] [02miraheze/CreateWiki] 07Universal-Omega 03f0e1f71 - Fix email notifications [18:19:31] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vpJTL [18:19:33] [02CreateWiki] 07Universal-Omega opened pull request 03#232: Fix email notifications - 13https://git.io/JBSl2 [18:20:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.61, 8.02, 7.42 [18:20:29] miraheze/CreateWiki - Universal-Omega the build passed. [18:22:20] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.15, 7.78, 7.41 [18:23:18] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JBS8r [18:23:19] [02miraheze/CreateWiki] 07Universal-Omega 03e01a2d2 - Update extension.json [18:23:21] [02CreateWiki] 07Universal-Omega synchronize pull request 03#232: Fix email notifications - 13https://git.io/JBSl2 [18:24:13] miraheze/CreateWiki - Universal-Omega the build passed. [18:24:20] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.32, 8.04, 7.55 [18:25:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBS4v [18:25:13] [02miraheze/services] 07MirahezeSSLBot 030feb61e - BOT: Updating services config for wikis [18:26:20] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.96, 7.35, 7.36 [18:32:20] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.61, 7.73, 7.51 [18:33:32] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.35, 7.72, 7.58 [18:34:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.24, 7.16, 7.33 [18:35:35] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.45, 7.37, 7.48 [18:37:32] PROBLEM - wiki.usagihime.ml - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.usagihime.ml could not be found [18:37:34] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.32, 8.23, 7.79 [18:37:35] PROBLEM - stardewmodding.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for stardewmodding.wiki could not be found [18:39:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.88, 7.63, 7.62 [18:42:22] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.01, 6.00, 6.70 [18:44:12] RECOVERY - wiki.usagihime.ml - reverse DNS on sslhost is OK: SSL OK - wiki.usagihime.ml reverse DNS resolves to cp13.miraheze.org - CNAME OK [18:44:15] RECOVERY - stardewmodding.wiki - reverse DNS on sslhost is OK: SSL OK - stardewmodding.wiki reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [18:45:35] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.45, 7.10, 7.28 [18:49:34] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.07, 7.08, 7.27 [18:51:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.32, 7.61, 7.43 [18:54:28] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.58, 2.27, 1.54 [18:55:33] [02mw-config] 07scrapter opened pull request 03#4033: Add $wgMinervaTalkAtTop to isvwiki - 13https://git.io/JBSzT [18:56:26] miraheze/mw-config - scrapter the build passed. [18:56:26] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.79, 1.69, 1.41 [18:57:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.85, 7.28, 7.43 [19:00:25] [02mw-config] 07scrapter commented on pull request 03#4033: Add $wgMinervaTalkAtTop to isvwiki - 13https://git.io/JBSzH [19:00:29] PROBLEM - mail2 APT on mail2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:03:00] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.97, 7.66, 7.07 [19:03:13] [02mw-config] 07scrapter deleted a comment on pull request 03#4033: Add $wgMinervaTalkAtTop to isvwiki - 13https://git.io/JBSzH [19:03:31] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.74, 7.68, 7.52 [19:04:48] [02mw-config] 07scrapter synchronize pull request 03#4033: Add $wgMinervaTalkAtTop to isvwiki - 13https://git.io/JBSzT [19:04:56] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.96, 7.66, 7.17 [19:05:05] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:05:21] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:05:25] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:05:46] miraheze/mw-config - scrapter the build passed. [19:05:57] PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:06:13] PROBLEM - jobrunner4 APT on jobrunner4 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:06:15] PROBLEM - jobrunner3 APT on jobrunner3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:06:45] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:07:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.92, 7.83, 7.67 [19:07:37] [02mw-config] 07scrapter synchronize pull request 03#4033: Add $wgMinervaTalkAtTop to isvwiki - 13https://git.io/JBSzT [19:08:29] miraheze/mw-config - scrapter the build passed. [19:09:50] [02CreateWiki] 07Universal-Omega closed pull request 03#232: Fix email notifications - 13https://git.io/JBSl2 [19:09:51] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JBS2d [19:09:53] [02miraheze/CreateWiki] 07Universal-Omega 031123fb8 - Fix email notifications (#232) [19:09:54] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vpJTL [19:09:56] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [19:10:45] miraheze/CreateWiki - Universal-Omega the build passed. [19:10:48] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.38, 7.47, 7.13 [19:11:40] [02mw-config] 07scrapter synchronize pull request 03#4033: Add $wgMinervaTalkAtTop to isvwiki - 13https://git.io/JBSzT [19:12:02] [02mw-config] 07scrapter commented on pull request 03#4033: Add $wgMinervaTalkAtTop to isvwiki - 13https://git.io/JBSaT [19:12:18] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JBSaY [19:12:20] [02miraheze/CreateWiki] 07Universal-Omega 03b9f57d8 - Add bundling to Echo notifications for email digest support [19:12:21] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vpJTL [19:12:23] [02CreateWiki] 07Universal-Omega opened pull request 03#233: Add bundling to Echo notifications for email digest support - 13https://git.io/JBSa3 [19:12:35] miraheze/mw-config - scrapter the build passed. [19:12:44] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.64, 7.04, 7.01 [19:12:54] Finally it worked! 🎉 [19:13:22] miraheze/CreateWiki - Universal-Omega the build passed. [19:13:29] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.89, 7.91, 7.64 [19:13:42] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JBSa2 [19:13:43] [02miraheze/CreateWiki] 07Universal-Omega 0308fb9b4 - Update extension.json [19:13:45] [02CreateWiki] 07Universal-Omega synchronize pull request 03#233: Add bundling to Echo notifications for email digest support - 13https://git.io/JBSa3 [19:14:39] miraheze/CreateWiki - Universal-Omega the build passed. [19:14:42] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.27, 7.93, 7.34 [19:15:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.41, 7.70, 7.60 [19:15:40] [02mw-config] 07Universal-Omega reviewed pull request 03#4033 commit - 13https://git.io/JBSax [19:16:39] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.04, 7.41, 7.21 [19:17:18] [02mw-config] 07scrapter reviewed pull request 03#4033 commit - 13https://git.io/JBSVl [19:17:54] [02mw-config] 07Universal-Omega reviewed pull request 03#4033 commit - 13https://git.io/JBSVu [19:18:54] [02mw-config] 07scrapter reviewed pull request 03#4033 commit - 13https://git.io/JBSVM [19:23:35] [02mw-config] 07scrapter synchronize pull request 03#4033: Add $wgMinervaTalkAtTop to isvwiki - 13https://git.io/JBSzT [19:23:59] [02mw-config] 07scrapter reviewed pull request 03#4033 commit - 13https://git.io/JBSwE [19:24:25] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.18, 6.29, 6.71 [19:24:31] miraheze/mw-config - scrapter the build passed. [19:25:25] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.19, 7.62, 7.54 [19:25:41] [02mw-config] 07Universal-Omega reviewed pull request 03#4033 commit - 13https://git.io/JBSwM [19:27:25] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.59, 7.09, 7.35 [19:28:15] [02mw-config] 07scrapter synchronize pull request 03#4033: Add $wgMinervaTalkAtTop to isvwiki - 13https://git.io/JBSzT [19:29:25] [02mw-config] 07scrapter reviewed pull request 03#4033 commit - 13https://git.io/JBSrY [19:29:28] [02mw-config] 07Universal-Omega reviewed pull request 03#4033 commit - 13https://git.io/JBSr3 [19:30:12] miraheze/mw-config - scrapter the build passed. [19:32:34] [02mw-config] 07scrapter synchronize pull request 03#4033: Add $wgMinervaTalkAtTop to isvwiki - 13https://git.io/JBSzT [19:33:18] [02mw-config] 07Universal-Omega reviewed pull request 03#4033 commit - 13https://git.io/JBSrN [19:33:28] miraheze/mw-config - scrapter the build passed. [19:33:53] [02mw-config] 07scrapter synchronize pull request 03#4033: Add $wgMinervaTalkAtTop to isvwiki - 13https://git.io/JBSzT [19:33:54] [02mw-config] 07scrapter synchronize pull request 03#4033: Add $wgMinervaTalkAtTop to isvwiki - 13https://git.io/JBSzT [19:34:49] miraheze/mw-config - scrapter the build passed. [19:35:07] miraheze/mw-config - scrapter the build passed. [19:35:14] [02mw-config] 07scrapter reviewed pull request 03#4033 commit - 13https://git.io/JBSoG [19:38:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.50, 7.12, 6.72 [19:47:37] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.68, 6.30, 6.76 [19:54:20] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.54, 7.54, 7.06 [19:55:39] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.36, 7.58, 7.09 [19:57:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.81, 7.25, 7.03 [20:00:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.66, 7.31, 7.25 [20:03:48] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.28, 19.92, 18.35 [20:04:26] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.41, 8.05, 7.55 [20:05:47] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.96, 19.36, 18.35 [20:06:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.20, 7.79, 7.52 [20:07:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.93, 7.77, 7.26 [20:08:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.21, 8.21, 7.69 [20:09:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.35, 7.58, 7.25 [20:11:41] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.08, 7.89, 7.39 [20:12:28] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JBSXS [20:12:29] [02miraheze/ManageWiki] 07Universal-Omega 0305a63a1 - Update en.json [20:12:31] [02ManageWiki] 07Universal-Omega synchronize pull request 03#296: Add help-message to 'description' field - 13https://git.io/JByFu [20:13:21] miraheze/ManageWiki - Universal-Omega the build passed. [20:13:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.53, 7.21, 7.19 [20:16:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.56, 7.96, 7.89 [20:17:39] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.07, 7.23, 7.18 [20:18:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.41, 7.96, 7.88 [20:20:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.83, 7.65, 7.79 [20:24:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.39, 8.33, 7.95 [20:29:37] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.95, 7.80, 7.71 [20:31:36] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.04, 8.25, 7.86 [20:36:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.45, 7.50, 7.96 [20:40:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.56, 8.71, 8.30 [20:41:59] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 243.32 ms [20:42:50] @Bukkit@localhost: can you stick to one username on github pls? [20:43:29] [02RemovePII] 07Universal-Omega commented on pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/JBSyH [20:44:20] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.21, 7.64, 7.95 [20:49:32] Universal_Omega: to the JS point yes [20:49:34] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.81, 7.03, 7.70 [20:50:07] You got a risk otherwise of malicious impact either causing a vanishing or revealing info on it [20:51:23] RhinosF1: yeah that's why I was thinking about it. Doesn't hurt to add it regardless so I will add the disallowUserJs thing. [20:51:41] Good [20:52:01] I think RemovePII is a sensitive page [20:54:31] RhinosF1: Yeah it doesn't even allow viewing it without a specific right. It's not sensitive just to view, but to submit it, it's is extremely dangerous. And that's where JS comes in at, don't want any JS logging that data on submit which I do believe is possible. But maybe isn't unless you have the right but I'd rather be safe. [20:55:23] There's a bunch of checks to make sure things are safe to submit it also. And as much as I can think of to verify it'll succeed. [20:55:37] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.15, 7.21, 7.47 [20:55:41] Yeah exactly [20:56:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.25, 7.08, 7.25 [20:57:36] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.98, 7.38, 7.49 [20:58:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.39, 6.93, 7.18 [21:07:46] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.11, 21.41, 19.28 [21:08:17] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 6.00, 4.20, 2.99 [21:09:45] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.76, 19.58, 18.87 [21:10:16] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.87, 3.26, 2.80 [21:12:20] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.88, 6.06, 6.71 [21:13:46] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.70, 5.81, 6.62 [21:20:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.06, 7.17, 6.91 [21:22:21] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.11, 6.48, 6.69 [21:23:33] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBS5p [21:23:35] [02miraheze/RemovePII] 07Universal-Omega 036dedf2e - Add additional checks [21:23:36] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [21:24:27] miraheze/RemovePII - Universal-Omega the build passed. [21:37:49] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JBSFb [21:37:50] [02miraheze/RemovePII] 07Universal-Omega 0369f79d1 - Update SpecialGenerateRandomHash.php [21:37:52] [02RemovePII] 07Universal-Omega synchronize pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/Js0PN [21:38:50] miraheze/RemovePII - Universal-Omega the build passed. [21:42:50] [02RemovePII] 07Universal-Omega edited a comment on pull request 03#3: Create SpecialRemovePII (T7286: phase two and phase three) - 13https://git.io/JBSyH [21:54:22] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.33, 1.76, 1.28 [21:56:22] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.39, 1.56, 1.27 [21:58:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.86, 6.66, 7.95 [22:02:39] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.65, 7.35, 7.90 [22:10:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.08, 7.17, 7.85 [22:12:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.99, 7.09, 7.88 [22:16:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.46, 7.41, 7.80 [22:18:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.80, 7.29, 7.70 [22:26:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.25, 6.06, 6.77 [22:32:23] PROBLEM - cp12 Disk Space on cp12 is WARNING: DISK WARNING - free space: / 4237 MB (10% inode=96%); [22:34:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.38, 7.61, 7.27 [22:36:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.08, 6.68, 6.98 [22:36:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.10, 6.97, 6.80 [22:38:28] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.81, 5.89, 6.64 [22:38:40] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.10, 6.50, 6.63 [22:45:22] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.22, 7.11, 6.89 [22:46:07] !log sudo -u www-data php /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/rebuildVersionCache.php --wiki=loginwiki --save-gitinfo on mw* / jbr* / test3 [22:46:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:47:18] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.38, 6.54, 6.71 [22:50:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.01, 19.45, 17.20 [22:51:13] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.67, 7.55, 7.06 [22:51:30] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.00, 7.16, 6.78 [22:52:43] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.20, 19.33, 17.44 [22:53:10] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.18, 6.92, 6.88 [22:55:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.25, 6.43, 6.71 [22:55:55] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JB9Je [22:55:56] [02miraheze/MirahezeMagic] 07Universal-Omega 038beebba - test rebuildVersionCache with mediawiki-staging directory [22:55:58] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/fQRGX [22:56:03] [02MirahezeMagic] 07Universal-Omega opened pull request 03#275: test rebuildVersionCache with mediawiki-staging directory - 13https://git.io/JB9Jv [22:56:58] miraheze/MirahezeMagic - Universal-Omega the build passed. [22:57:15] !log [universalomega@test3] starting deploy of {'world': True} to skip [22:57:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:58:06] !log [universalomega@test3] finished deploy of {'world': True} to skip [22:58:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:58:52] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 25 minutes ago with 0 failures [22:59:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.74, 6.49, 6.66 [23:02:44] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JB9JN [23:02:45] [02miraheze/MirahezeMagic] 07Universal-Omega 035b88585 - Update rebuildVersionCache.php [23:02:46] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#275: test rebuildVersionCache with mediawiki-staging directory - 13https://git.io/JB9Jv [23:03:02] !log [universalomega@test3] starting deploy of {'world': True} to skip [23:03:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:03:15] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.12, 7.29, 6.95 [23:03:29] !log [universalomega@test3] finished deploy of {'world': True} to skip [23:03:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:03:39] miraheze/MirahezeMagic - Universal-Omega the build passed. [23:03:57] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.78, 8.07, 7.25 [23:07:48] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.81, 7.46, 7.25 [23:11:41] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.71, 6.14, 6.77 [23:15:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.23, 6.61, 6.89 [23:17:31] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.11, 6.13, 6.67 [23:21:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.74, 7.39, 7.12 [23:24:40] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.32, 7.85, 7.84 [23:26:07] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JB9IZ [23:26:08] [02miraheze/MirahezeMagic] 07Universal-Omega 03405ab4b - Update rebuildVersionCache.php [23:26:10] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#275: test rebuildVersionCache with mediawiki-staging directory - 13https://git.io/JB9Jv [23:26:24] !log [universalomega@test3] starting deploy of {'world': True} to skip [23:26:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:26:48] !log [universalomega@test3] finished deploy of {'world': True} to skip [23:26:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:01] miraheze/MirahezeMagic - Universal-Omega the build passed. [23:28:41] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.39, 8.25, 7.92 [23:29:51] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JB9Id [23:29:53] [02miraheze/MirahezeMagic] 07Universal-Omega 039ad290b - Update rebuildVersionCache.php [23:29:54] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#275: test rebuildVersionCache with mediawiki-staging directory - 13https://git.io/JB9Jv [23:30:05] !log [universalomega@test3] starting deploy of {'world': True} to skip [23:30:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:26] !log [universalomega@test3] finished deploy of {'world': True} to skip [23:30:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:47] miraheze/MirahezeMagic - Universal-Omega the build passed. [23:31:10] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.52, 7.70, 7.32 [23:33:01] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JB9La [23:33:03] [02miraheze/MirahezeMagic] 07Universal-Omega 03c7c09c1 - Update rebuildVersionCache.php [23:33:04] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#275: test rebuildVersionCache with mediawiki-staging directory - 13https://git.io/JB9Jv [23:33:15] !log [universalomega@test3] starting deploy of {'world': True} to skip [23:33:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:33:45] !log [universalomega@test3] finished deploy of {'world': True} to skip [23:33:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:33:59] miraheze/MirahezeMagic - Universal-Omega the build passed. [23:34:33] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.15, 6.63, 5.84 [23:36:29] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.53, 6.37, 5.83 [23:36:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.48, 20.05, 18.79 [23:38:41] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.12, 19.62, 18.77 [23:42:42] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.47, 6.69, 7.65 [23:42:49] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.23, 7.67, 7.97 [23:58:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.56, 6.25, 6.77