[00:02:49] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.00, 9.32, 9.48 [00:04:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.10, 9.27, 8.96 [00:04:38] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:05:12] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.11, 9.03, 8.79 [00:07:12] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.77, 9.42, 9.14 [00:08:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.80, 8.73, 8.70 [00:10:12] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.59, 10.36, 9.55 [00:11:15] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.66, 8.01, 8.45 [00:13:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.42, 9.89, 9.55 [00:13:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.63, 3.39, 3.46 [00:13:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:16:25] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.72, 3.11, 3.35 [00:17:55] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.07, 10.53, 9.85 [00:18:13] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.83, 10.52, 9.29 [00:19:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.50, 10.75, 9.94 [00:19:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.67, 8.09, 8.34 [00:20:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:26:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.22, 8.25, 7.81 [00:27:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.32, 8.72, 8.48 [00:31:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/8b0a8fc943e4 [00:31:14] [02miraheze/mw-config] 07Universal-Omega 038b0a8fc - only enable c4 on betaheze [00:31:15] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [00:31:17] [02mw-config] 07Universal-Omega opened pull request 03#4455: only enable c4 on betaheze - 13https://github.com/miraheze/mw-config/pull/4455 [00:31:53] [02mw-config] 07Universal-Omega closed pull request 03#4455: only enable c4 on betaheze - 13https://github.com/miraheze/mw-config/pull/4455 [00:31:54] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [00:31:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1df5a5dd6241...bcb54624a1c6 [00:31:57] [02miraheze/mw-config] 07Universal-Omega 03bcb5462 - only enable c4 on betaheze (#4455) [00:31:59] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [00:32:10] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'force': True} to skip [00:32:12] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'force': True} to skip - SUCCESS in 1s [00:32:16] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.00, 7.70, 7.81 [00:32:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:20] miraheze/mw-config - Universal-Omega the build passed. [00:32:25] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:32:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:59] miraheze/mw-config - Universal-Omega the build passed. [00:35:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.95, 4.19, 3.67 [00:36:24] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.20, 8.47, 8.47 [00:38:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.67, 3.54, 3.52 [00:45:58] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2550 bytes in 0.061 second response time [00:46:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.09, 10.11, 8.88 [00:47:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.80, 9.29, 8.94 [00:48:15] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.97, 3.11, 3.33 [00:49:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.60, 9.92, 9.05 [00:50:35] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.08, 9.61, 9.15 [00:51:21] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [00:51:26] !log [universalomega@mwtask111] DEPLOY ABORTED: Canary check failed for localhost [00:51:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:51:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:52:52] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [00:52:57] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.34, 10.34, 9.38 [00:53:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:53:06] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 13s [00:53:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:53:41] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.32, 9.32, 9.14 [00:54:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.37, 1.84, 1.62 [00:55:48] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/ff58e2e4f555 [00:55:49] [02miraheze/CreateWiki] 07Universal-Omega 03ff58e2e - add CreateWikiDatabaseSuffix config [00:55:51] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/CreateWiki [00:55:52] [02CreateWiki] 07Universal-Omega opened pull request 03#282: add CreateWikiDatabaseSuffix config - 13https://github.com/miraheze/CreateWiki/pull/282 [00:56:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.16, 9.47, 9.23 [00:56:41] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/ff58e2e4f555...2bf7262f0c70 [00:56:42] [02miraheze/CreateWiki] 07Universal-Omega 032bf7262 - Update extension.json [00:56:44] [02CreateWiki] 07Universal-Omega synchronize pull request 03#282: add CreateWikiDatabaseSuffix config - 13https://github.com/miraheze/CreateWiki/pull/282 [00:57:50] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.42, 1.69, 1.61 [00:58:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/85c91829a4e8 [00:58:02] [02miraheze/mw-config] 07Universal-Omega 0385c9182 - configure wgCreateWikiDatabaseSuffix [00:58:03] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [00:58:05] [02mw-config] 07Universal-Omega opened pull request 03#4456: configure wgCreateWikiDatabaseSuffix - 13https://github.com/miraheze/mw-config/pull/4456 [00:59:07] [02mw-config] 07Universal-Omega closed pull request 03#4456: configure wgCreateWikiDatabaseSuffix - 13https://github.com/miraheze/mw-config/pull/4456 [00:59:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/bcb54624a1c6...9600ec651c0b [00:59:10] [02miraheze/mw-config] 07Universal-Omega 039600ec6 - configure wgCreateWikiDatabaseSuffix (#4456) [00:59:11] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [00:59:13] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [00:59:15] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'force': True} to skip [00:59:17] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'force': True} to skip - SUCCESS in 0s [00:59:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:59:25] miraheze/mw-config - Universal-Omega the build passed. [00:59:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:59:48] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.92, 9.90, 9.37 [00:59:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.72, 8.66, 8.31 [00:59:51] !log [universalomega@test101] starting deploy of {'force': True, 'folders': 'w/extensions/CreateWiki'} to skip [00:59:52] !log [universalomega@test101] finished deploy of {'force': True, 'folders': 'w/extensions/CreateWiki'} to skip - SUCCESS in 0s [01:00:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:00:24] miraheze/mw-config - Universal-Omega the build passed. [01:00:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:02:41] miraheze/CreateWiki - Universal-Omega the build passed. [01:02:50] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.87, 9.81, 9.42 [01:03:17] [02CreateWiki] 07Universal-Omega closed pull request 03#282: add CreateWikiDatabaseSuffix config - 13https://github.com/miraheze/CreateWiki/pull/282 [01:03:18] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/c7ab79eb6a0d...8fa21869fe5b [01:03:20] [02miraheze/CreateWiki] 07Universal-Omega 038fa2186 - add CreateWikiDatabaseSuffix config (#282) [01:03:21] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/CreateWiki [01:03:23] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [01:05:47] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.11, 7.98, 8.14 [01:08:34] miraheze/CreateWiki - Universal-Omega the build passed. [01:13:05] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/0562087fea60...6ce630e3f10e [01:13:06] [02miraheze/mediawiki] 07Universal-Omega 036ce630e - Update CreateWiki [01:16:25] !log [universalomega@test101] starting deploy of {'world': True, 'force': True} to skip [01:16:39] !log [universalomega@mwtask111] starting deploy of {'world': True, 'force': True} to all [01:16:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:16:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:20:00] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.90, 7.94, 8.45 [01:20:19] !log [universalomega@test101] finished deploy of {'world': True, 'force': True} to skip - SUCCESS in 233s [01:20:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:23:20] !log [universalomega@mwtask111] finished deploy of {'world': True, 'force': True} to all - SUCCESS in 400s [01:23:25] !log [@mwtask111] starting deploy of {'config': True} to all [01:23:26] !log [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost [01:23:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:23:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:04] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.80, 3.73, 3.16 [01:27:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.96, 8.58, 8.57 [01:29:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.67, 3.23 [01:29:37] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [01:30:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.87, 9.80, 9.02 [01:30:43] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.07, 9.56, 9.13 [01:31:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.01, 9.35, 8.77 [01:32:58] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [01:33:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:33:10] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 12s [01:33:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:34:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.48, 9.58, 8.97 [01:36:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.91, 9.85, 9.41 [01:36:55] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.65, 9.69, 9.36 [01:38:15] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.90, 3.15, 3.29 [01:39:52] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.38, 9.64, 9.37 [01:40:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/9e0727a4b8d4 [01:40:08] [02miraheze/mw-config] 07Universal-Omega 039e0727a - Update cookies config for betaheze [01:40:09] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [01:40:11] [02mw-config] 07Universal-Omega opened pull request 03#4457: Update cookies config for betaheze - 13https://github.com/miraheze/mw-config/pull/4457 [01:40:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.20, 9.46, 9.07 [01:40:54] [02mw-config] 07Universal-Omega closed pull request 03#4457: Update cookies config for betaheze - 13https://github.com/miraheze/mw-config/pull/4457 [01:40:56] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [01:40:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9600ec651c0b...f56a3b4b553c [01:40:59] [02miraheze/mw-config] 07Universal-Omega 03f56a3b4 - Update cookies config for betaheze (#4457) [01:41:00] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [01:41:11] miraheze/mw-config - Universal-Omega the build passed. [01:41:13] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'force': True} to skip [01:41:15] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'force': True} to skip - SUCCESS in 1s [01:41:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:41:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:41:56] miraheze/mw-config - Universal-Omega the build passed. [01:42:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.44, 9.28, 9.33 [01:44:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/2b8b702c15b0 [01:44:48] [02miraheze/mw-config] 07Universal-Omega 032b8b702 - expand setting WANCache for *.betaheze.org [01:44:49] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [01:44:51] [02mw-config] 07Universal-Omega opened pull request 03#4458: expand setting WANCache for *.betaheze.org - 13https://github.com/miraheze/mw-config/pull/4458 [01:45:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.50, 9.83, 9.51 [01:45:51] miraheze/mw-config - Universal-Omega the build passed. [01:45:55] [02mw-config] 07Universal-Omega closed pull request 03#4458: expand setting WANCache for *.betaheze.org - 13https://github.com/miraheze/mw-config/pull/4458 [01:45:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f56a3b4b553c...c786c334355b [01:45:58] [02miraheze/mw-config] 07Universal-Omega 03c786c33 - expand setting WANCache for *.betaheze.org (#4458) [01:45:59] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'force': True} to skip [01:46:00] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [01:46:00] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'force': True} to skip - SUCCESS in 0s [01:46:01] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [01:46:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:46:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.33, 8.76, 8.94 [01:47:04] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'ignoretime': True} to skip [01:47:05] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'ignoretime': True} to skip - SUCCESS in 0s [01:47:16] miraheze/mw-config - Universal-Omega the build passed. [01:47:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 3.59, 7.63, 9.26 [01:47:27] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 22201 bytes in 1.005 second response time [01:48:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.95, 9.22, 9.31 [01:49:32] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.34, 9.16, 9.05 [01:50:16] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 0.69, 4.64, 7.83 [01:50:19] PROBLEM - mw112 MediaWiki Rendering on mw112 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2550 bytes in 0.102 second response time [01:50:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.96, 8.52, 9.59 [01:51:51] !log [@mwtask111] starting deploy of {'config': True} to all [01:51:56] !log [@mwtask111] DEPLOY ABORTED: Canary check failed for mw112 [01:52:09] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 22230 bytes in 0.369 second response time [01:52:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.31, 8.01, 8.64 [01:53:21] getting havoc on the SN and apparently various 404 errors in other regions of miraheze [01:55:11] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.76, 7.18, 8.22 [01:55:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:56:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.63, 6.73, 8.44 [01:57:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.18, 7.45, 8.31 [01:58:51] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.836 second response time [01:59:00] raidarr: fixed (for now.....) I am working on a better fix so it won't happen again. [01:59:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.20, 6.94, 8.07 [02:04:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.28, 9.10, 9.82 [02:04:45] !log DELETE FROM mw_settings WHERE s_dbname='betawiki'; on mhglobal [02:04:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:05:11] !log DELETE FROM mw_permissions WHERE perm_dbname='betawiki'; on mhglobal [02:05:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:05:31] !log DELETE FROM mw_namespaces WHERE ns_dbname='betawiki'; on mhglobal [02:05:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:06:29] !log DELETE FROM cw_wikis WHERE wiki_dbname='betawiki'; on mhglobal [02:06:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:07:09] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.27, 9.42, 9.80 [02:09:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:10:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.63, 8.73, 9.45 [02:12:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.53, 8.30, 8.36 [02:12:54] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.08, 9.54, 9.63 [02:14:23] !log DELETE FROM cw_wikis WHERE wiki_dbname='testcreatebetawikiwiki'; on testglobal [02:14:29] !log DELETE FROM cw_wikis WHERE wiki_dbname='testcreatebetawiki2wiki'; on testglobal [02:14:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:14:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:16:12] PROBLEM - test101 MediaWiki Rendering on test101 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2550 bytes in 0.109 second response time [02:18:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.07, 9.17, 7.74 [02:18:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.26, 9.60, 8.83 [02:18:48] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.57, 9.39, 9.61 [02:21:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.35, 9.79, 8.27 [02:21:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.28, 9.28, 8.91 [02:22:29] !log [universalomega@test101] starting deploy of {'config': True} to skip [02:22:30] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for localhost [02:22:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:22:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:24:02] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [02:24:03] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.07, 9.76, 8.50 [02:24:24] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.52, 7.91, 8.46 [02:24:58] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.21, 3.82, 2.16 [02:27:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.71, 9.64, 8.67 [02:27:40] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.43, 10.42, 9.91 [02:28:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.65, 2.38, 1.87 [02:29:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:31:24] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.33, 1.77, 1.72 [02:33:58] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.88, 1.78, 1.77 [02:34:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.79, 1.33, 1.55 [02:34:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:36:54] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.11, 1.59, 1.70 [02:37:04] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.44, 9.62, 8.86 [02:39:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:40:01] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.98, 9.19, 8.85 [02:40:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/31bae3c349ee [02:40:32] [02miraheze/mw-config] 07Universal-Omega 0331bae3c - disable `$wgCreateWikiUseJobQueue` on betaheze [02:40:33] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [02:40:35] [02mw-config] 07Universal-Omega opened pull request 03#4459: disable `$wgCreateWikiUseJobQueue` on betaheze - 13https://github.com/miraheze/mw-config/pull/4459 [02:40:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c786c334355b...3fdc0e297f42 [02:40:44] [02miraheze/mw-config] 07Universal-Omega 033fdc0e2 - disable `$wgCreateWikiUseJobQueue` on betaheze (#4459) [02:40:46] [02mw-config] 07Universal-Omega closed pull request 03#4459: disable `$wgCreateWikiUseJobQueue` on betaheze - 13https://github.com/miraheze/mw-config/pull/4459 [02:40:47] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [02:40:49] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [02:41:04] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'force': True} to skip [02:41:05] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'force': True} to skip - SUCCESS in 1s [02:41:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:41:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:41:30] miraheze/mw-config - Universal-Omega the build passed. [02:41:53] miraheze/mw-config - Universal-Omega the build passed. [02:45:43] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.26, 8.04, 8.41 [02:46:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:47:18] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.47, 8.10, 8.38 [02:51:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:52:03] !log [universalomega@test101] sudo -u www-data php /srv/mediawiki/w/maintenance/createAndPromote.php --wiki=smwwikibeta --bureaucrat --sysop --force Universal Omega (END - exit=256) [02:52:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:20] !log [@mwtask111] starting deploy of {'config': True} to all [02:52:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:34] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 13s [02:52:39] !log [universalomega@test101] sudo -u www-data php /srv/mediawiki/w/maintenance/createAndPromote.php --wiki=smwwikibeta --bureaucrat --sysop --force Universal_Omega (END - exit=0) [02:52:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:53:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.22, 9.27, 9.89 [02:55:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/923849d6c399 [02:55:14] [02miraheze/mw-config] 07Universal-Omega 03923849d - add globa lgroup management to bureaucrat on betawiki [02:55:15] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [02:55:20] [02mw-config] 07Universal-Omega opened pull request 03#4460: add global group management to bureaucrat on betawiki - 13https://github.com/miraheze/mw-config/pull/4460 [02:56:20] miraheze/mw-config - Universal-Omega the build passed. [02:56:27] [02mw-config] 07Universal-Omega closed pull request 03#4460: add global group management to bureaucrat on betawiki - 13https://github.com/miraheze/mw-config/pull/4460 [02:56:27] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.35, 9.86, 9.98 [02:56:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3fdc0e297f42...278d260ac136 [02:56:30] [02miraheze/mw-config] 07Universal-Omega 03278d260 - add global group management to bureaucrat on betawiki (#4460) [02:56:31] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [02:56:33] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [02:56:42] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'force': True} to skip [02:56:43] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'force': True} to skip - SUCCESS in 0s [02:56:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:57:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:57:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.93, 8.90, 8.70 [02:57:35] miraheze/mw-config - Universal-Omega the build passed. [02:57:46] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'ignoretime': True} to skip [02:57:47] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'ignoretime': True} to skip - SUCCESS in 0s [02:57:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:57:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:58:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:59:13] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.02, 8.35, 8.02 [02:59:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.63, 9.58, 9.86 [02:59:50] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.09, 2.27, 2.06 [03:00:21] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.52, 8.03, 8.42 [03:02:24] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.28, 7.96, 7.90 [03:02:28] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.90, 10.79, 10.32 [03:02:59] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.43, 1.77, 1.89 [03:07:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.87, 9.52, 8.93 [03:07:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.50, 9.70, 9.11 [03:09:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.62, 2.05, 1.67 [03:10:17] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.97, 10.37, 9.45 [03:12:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.12, 8.66, 8.26 [03:12:49] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.37, 1.31, 1.45 [03:15:10] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.29, 8.48, 8.26 [03:16:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.97, 9.14, 9.30 [03:17:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.76, 9.07, 9.72 [03:19:02] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.75, 9.69, 9.21 [03:20:04] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.09, 9.45, 9.76 [03:20:59] [02CreateWiki] 07lens0021 opened pull request 03#283: Use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283 [03:22:08] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.20, 8.66, 8.91 [03:22:08] [02CreateWiki] 07Universal-Omega commented on pull request 03#283: Use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283#issuecomment-1023850012 [03:22:30] !log [@mwtask111] starting deploy of {'config': True} to all [03:22:40] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 9s [03:22:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:22:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:23:03] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.66, 9.36, 9.68 [03:23:28] [02CreateWiki] 07lens0021 synchronize pull request 03#283: Use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283 [03:23:55] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.95, 1.33, 1.58 [03:24:57] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.95, 9.75, 9.47 [03:25:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.57, 8.53, 8.89 [03:25:09] [02CreateWiki] 07Universal-Omega edited a comment on pull request 03#283: Use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283#issuecomment-1023850012 [03:25:34] miraheze/CreateWiki - Universal-Omega the build has errored. [03:27:10] miraheze/CreateWiki - Universal-Omega the build has errored. [03:27:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.64, 8.91, 8.93 [03:28:02] [02CreateWiki] 07lens0021 commented on pull request 03#283: Use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283#issuecomment-1023852510 [03:28:34] [02CreateWiki] 07Universal-Omega commented on pull request 03#283: Use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283#issuecomment-1023852747 [03:29:39] miraheze/CreateWiki - Universal-Omega the build has errored. [03:30:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.36, 8.03, 8.59 [03:31:55] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.04, 9.79, 9.54 [03:32:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.74, 3.42, 3.35 [03:33:48] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.18, 7.68, 8.35 [03:35:48] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.82, 3.22, 3.28 [03:40:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.32, 8.73, 8.66 [03:42:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.22, 9.71, 9.16 [03:43:47] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.03, 9.86, 9.07 [03:45:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.72, 8.17, 8.64 [03:46:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.85, 8.63, 8.54 [03:46:19] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=pokefanonwiki --username-prefix=wikia:pokefanon --report=1 --no-updates /home/reception/esfakemon_pages_full.xml (END - exit=256) [03:46:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.81, 8.52, 8.66 [03:47:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:47:40] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=pokefanonwiki --username-prefix=wikia:pokefanon --report=1 --no-updates /home/reception/esfakemon_pages_full.xml (START) [03:48:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:51:34] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.02, 2.02, 1.53 [03:52:02] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.87, 8.65, 8.55 [03:52:21] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.27, 8.08, 8.29 [03:52:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.04, 7.48, 8.23 [03:54:32] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.86, 1.56, 1.43 [03:54:56] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.18, 8.32, 8.44 [03:57:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.84, 8.00, 7.53 [04:00:51] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.50, 7.76, 7.52 [04:03:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:08:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.53, 8.58, 8.32 [04:10:18] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.74, 8.89, 9.80 [04:11:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.67, 7.52, 7.98 [04:12:35] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.01, 8.92, 8.48 [04:13:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:18:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.68, 9.33, 8.72 [04:18:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:19:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.13, 9.61, 9.71 [04:21:28] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.29, 9.94, 9.35 [04:22:10] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.66, 9.84, 9.79 [04:23:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:24:19] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.73, 9.96, 9.15 [04:24:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.43, 10.60, 9.75 [04:27:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.51, 9.72, 9.57 [04:29:57] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 7.37, 7.29, 5.56 [04:30:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.05, 9.56, 9.37 [04:31:10] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.36, 9.40, 9.45 [04:31:10] [02CreateWiki] 07lens0021 synchronize pull request 03#283: Use AutoloadNamespaces - 13https://github.com/miraheze/CreateWiki/pull/283 [04:33:11] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.17, 10.52, 9.75 [04:34:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.05, 8.55, 9.11 [04:35:42] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 8.54, 7.89, 6.35 [04:36:18] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.42, 9.57, 9.50 [04:36:54] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.31, 8.99, 9.17 [04:38:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:39:13] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.17, 8.94, 9.27 [04:42:07] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.23, 9.34, 9.35 [04:43:47] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.47, 8.30, 8.00 [04:45:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.51, 9.38, 9.35 [04:45:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:45:40] miraheze/CreateWiki - Universal-Omega the build has errored. [04:47:17] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 7.17, 7.70, 7.19 [04:49:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.61, 9.24, 8.54 [04:49:40] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.76, 7.66, 7.86 [04:53:10] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 8.62, 7.98, 7.42 [04:53:50] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.44, 9.74, 9.53 [04:54:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.73, 9.58, 9.76 [04:54:25] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [04:55:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.40, 8.47, 8.44 [04:56:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.34, 9.35, 9.40 [05:05:52] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.21, 7.16, 7.95 [05:08:34] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.10, 7.68, 8.43 [05:11:52] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.19, 7.36, 8.38 [05:16:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.73, 8.53, 9.60 [05:20:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:21:31] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:28:44] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.40, 10.01, 9.88 [05:35:29] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.47, 6.50, 6.74 [05:36:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:49:55] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.56, 1.98, 1.41 [05:52:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.79, 9.33, 9.87 [05:52:54] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.46, 1.38, 1.28 [05:57:03] [02CreateWiki] 07Universal-Omega edited pull request 03#281: [Performance] only connect to database if actually used in CreateWikiJson - 13https://github.com/miraheze/CreateWiki/pull/281 [05:57:45] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.29, 1.86, 1.27 [05:58:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.07, 9.49, 9.69 [06:00:54] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.68, 1.41, 1.21 [06:04:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.69, 9.84, 9.86 [06:07:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 9.69, 10.36, 10.10 [06:10:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.08, 9.31, 9.74 [06:11:35] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:18:35] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:21:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.82, 9.96, 9.00 [06:22:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.61, 8.93, 9.05 [06:24:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.38, 9.48, 8.99 [06:25:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.60, 8.84, 9.01 [06:27:45]