[00:00:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.68, 3.80, 3.62 [00:02:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.91, 3.70, 3.17 [00:03:27] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.04, 3.85, 3.70 [00:04:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.87, 3.91, 3.60 [00:04:27] PROBLEM - mem121 Puppet on mem121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:06:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.74, 4.54, 3.99 [00:06:15] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [00:06:27] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 4.08, 3.71 [00:06:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.46, 3.37, 3.18 [00:08:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.45, 3.70, 3.74 [00:08:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.71, 4.00, 3.72 [00:14:12] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.41, 4.45, 3.93 [00:14:12] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.67, 1.35, 1.90 [00:14:27] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.57, 4.00, 3.73 [00:14:36] ok : [RESOLVED] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:16:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.16, 3.79, 3.75 [00:16:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.27, 3.75, 3.67 [00:18:08] !log [@test101] starting deploy of {'config': True} to all [00:18:09] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [00:18:12] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 7.39, 2.63, 2.19 [00:18:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:19:16] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.89, 1.42, 1.14 [00:20:27] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.21, 2.98, 3.37 [00:21:13] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.02, 1.36, 1.16 [00:22:11] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.87, 1.65, 1.89 [00:22:17] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [00:24:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.60, 3.01, 3.38 [00:26:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.15, 3.29 [00:28:10] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.63, 1.10, 1.57 [00:32:27] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.85, 3.99, 3.62 [00:32:27] RECOVERY - mem121 Puppet on mem121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:33:27] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:34:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.31, 3.68, 3.55 [00:36:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.12, 3.98, 3.46 [00:36:27] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.42, 4.82, 3.99 [00:37:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.27, 4.81, 3.76 [00:41:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.09, 3.43, 3.46 [00:42:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.41, 3.42 [00:43:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.98, 3.28, 3.39 [00:43:59] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.29, 3.27, 3.36 [00:49:02] !log [@test101] starting deploy of {'config': True} to all [00:49:03] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [00:49:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:49:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:52:06] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.21, 1.99, 1.57 [00:52:17] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:53:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.42, 3.90, 3.54 [00:53:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 4.17, 3.69 [00:54:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:55:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.83, 3.69, 3.50 [00:57:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.89, 3.95, 3.74 [00:58:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.90, 2.00, 1.72 [00:59:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.50, 3.08, 3.30 [00:59:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.72, 4.04, 3.81 [01:00:05] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.90, 2.41, 1.91 [01:01:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.67, 3.71 [01:03:18] RECOVERY - test101 Check Gluster Clients on test101 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [01:04:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:06:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.13, 1.86, 1.85 [01:07:17] !log [universalomega@test101] starting deploy of {'config': True, 'ignoretime': True} to all [01:07:18] !log [universalomega@test101] finished deploy of {'config': True, 'ignoretime': True} to all - SUCCESS in 1s [01:07:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:07:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:08:04] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.27, 2.00, 1.90 [01:12:03] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.14, 1.68, 1.80 [01:13:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.72, 3.69, 3.64 [01:14:03] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 5.02, 2.45, 2.04 [01:17:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.08, 3.44, 3.28 [01:18:23] !log [@test101] starting deploy of {'config': True} to all [01:18:24] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [01:18:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:18:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:19:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.58, 3.11, 3.17 [01:19:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.83, 3.97, 3.83 [01:21:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.69, 4.08, 3.88 [01:25:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.43, 3.66, 3.41 [01:31:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.76, 3.17, 3.29 [01:33:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.95, 3.71, 3.94 [01:34:09] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [01:34:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d0387e334149...6d598bf84689 [01:34:12] [02miraheze/mw-config] 07Universal-Omega 036d598bf - fix [01:34:54] !log [universalomega@test101] starting deploy of {'config': True, 'ignoretime': True} to all [01:34:55] !log [universalomega@test101] finished deploy of {'config': True, 'ignoretime': True} to all - SUCCESS in 1s [01:34:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:35:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:35:22] miraheze/mw-config - Universal-Omega the build passed. [01:35:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.87, 3.97, 4.02 [01:36:21] FINALLY!!! Now it does seem to work right, with much better performance, and works on all wikis tested through proxy and otherwise! [01:37:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.38, 3.44, 3.82 [01:41:46] [02mediawiki] 07Universal-Omega closed pull request 03#6273: Bump extensions/CreateWiki from `d727020` to `151f351` - 13https://github.com/miraheze/mediawiki/pull/6273 [01:41:48] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/CreateWiki-151f351 [01:41:49] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/CreateWiki-151f351 - 13https://github.com/miraheze/mediawiki [01:41:51] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/8c551728482f...c2364ade75df [01:41:52] [02miraheze/mediawiki] 07dependabot[bot] 03c2364ad - Bump extensions/CreateWiki from `d727020` to `151f351` (#6273) [01:42:11] !log [universalomega@test101] starting deploy of {'world': True} to all [01:42:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:42:53] !log [universalomega@mwtask111] starting deploy of {'world': True} to all [01:43:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:43:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.54, 3.32 [01:43:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 3.63, 3.72 [01:45:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.56, 3.08, 3.17 [01:45:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.35, 3.21, 3.56 [01:46:20] !log [universalomega@test101] finished deploy of {'world': True} to all - SUCCESS in 248s [01:46:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:48:23] !log [@test101] starting deploy of {'config': True} to all [01:48:24] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [01:48:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:48:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:49:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.68, 3.82, 3.70 [01:51:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.94, 5.12, 4.08 [01:52:12] !log [universalomega@mwtask111] finished deploy of {'world': True} to all - SUCCESS in 559s [01:52:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:57:18] !log [universalomega@test101] starting deploy of {'config': True, 'ignoretime': True} to all [01:57:20] !log [universalomega@test101] finished deploy of {'config': True, 'ignoretime': True} to all - SUCCESS in 1s [01:57:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:57:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.87, 3.85 [01:57:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:57:57] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.91, 1.26, 1.92 [01:59:10] !log [universalomega@test101] starting deploy of {'config': True, 'ignoretime': True} to all [01:59:11] !log [universalomega@test101] finished deploy of {'config': True, 'ignoretime': True} to all - SUCCESS in 0s [01:59:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:59:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:59:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.14, 3.81, 3.95 [02:01:56] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.51, 0.89, 1.61 [02:07:00] PROBLEM - test101 MediaWiki Rendering on test101 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2550 bytes in 0.088 second response time [02:07:18] !log [universalomega@test101] starting deploy of {'config': True, 'ignoretime': True} to all [02:07:19] !log [universalomega@test101] finished deploy of {'config': True, 'ignoretime': True} to all - SUCCESS in 1s [02:07:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:07:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:08:58] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 51923 bytes in 1.345 second response time [02:12:10] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a080dd192629...a6a68ed2fbc8 [02:12:12] [02miraheze/ssl] 07MacFan4000 03a6a68ed - renew id.altilunium.xyz [02:17:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.47, 4.03, 3.72 [02:18:16] !log [@test101] starting deploy of {'config': True} to all [02:18:17] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [02:18:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:18:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:19:03] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/a439174459fd [02:19:04] [02miraheze/ssl] 07MacFan4000 03a439174 - remove idleon.info [02:19:06] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [02:19:07] [02ssl] 07MacFan4000 opened pull request 03#503: remove idleon.info - 13https://github.com/miraheze/ssl/pull/503 [02:19:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.26, 2.94, 3.30 [02:20:02] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a439174459fd...9b680fb14784 [02:20:04] [02miraheze/ssl] 07MacFan4000 039b680fb - Update certs.yaml [02:20:05] [02ssl] 07MacFan4000 synchronize pull request 03#503: remove idleon.info - 13https://github.com/miraheze/ssl/pull/503 [02:21:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.86, 3.75, 3.70 [02:21:39] [02ssl] 07MacFan4000 closed pull request 03#503: remove idleon.info - 13https://github.com/miraheze/ssl/pull/503 [02:21:40] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/a6a68ed2fbc8...3f817a868017 [02:21:41] [02miraheze/ssl] 07MacFan4000 033f817a8 - remove idleon.info (#503) [02:21:43] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [02:21:44] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [02:25:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.33, 3.65, 3.43 [02:25:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.33, 3.65, 3.43 [02:25:32] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/idleon.info.key] [02:27:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.87, 3.61, 3.43 [02:27:35] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/idleon.info.key] [02:27:56] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/private/idleon.info.key] [02:29:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.50, 3.06, 3.38 [02:29:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.57, 3.17, 3.29 [02:40:05] !log [universalomega@test101] starting deploy of {'config': True, 'ignoretime': True} to all [02:40:06] !log [universalomega@test101] finished deploy of {'config': True, 'ignoretime': True} to all - SUCCESS in 0s [02:40:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:40:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:40:46] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [02:40:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6d598bf84689...92f2f06c55b3 [02:40:49] [02miraheze/mw-config] 07Universal-Omega 0392f2f06 - add use of ObjectCache [02:41:01] !log [universalomega@test101] starting deploy of {'config': True, 'ignoretime': True} to all [02:41:02] !log [universalomega@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [02:41:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:41:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:42:00] miraheze/mw-config - Universal-Omega the build passed. [02:43:13] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [02:43:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/92f2f06c55b3...ad5a5bf4c429 [02:43:17] [02miraheze/mw-config] 07Universal-Omega 03ad5a5bf - Update MirahezeFunctions.php [02:43:30] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:44:20] miraheze/mw-config - Universal-Omega the build passed. [02:44:46] PROBLEM - test101 MediaWiki Rendering on test101 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2550 bytes in 0.802 second response time [02:48:25] !log [@test101] starting deploy of {'config': True} to all [02:48:26] !log [@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [02:48:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.92, 2.96, 3.89 [02:48:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:48:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:49:16] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [02:49:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ad5a5bf4c429...8bf03aa5faf5 [02:49:19] [02miraheze/mw-config] 07Universal-Omega 038bf03aa - Update MirahezeFunctions.php [02:50:27] miraheze/mw-config - Universal-Omega the build has errored. [02:50:46] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 52250 bytes in 0.697 second response time [02:50:52] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [02:55:34] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [02:55:55] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [02:56:27] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.37, 3.94, 3.89 [02:58:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.83, 3.12, 2.87 [02:58:38] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [02:58:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8bf03aa5faf5...dcf3155b92b0 [02:58:41] [02miraheze/mw-config] 07Universal-Omega 03dcf3155 - Update MirahezeFunctions.php [02:59:52] miraheze/mw-config - Universal-Omega the build passed. [03:01:10] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.19, 2.72, 2.76 [03:03:09] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [03:03:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/dcf3155b92b0...f5bb14c831a0 [03:03:12] [02miraheze/mw-config] 07Universal-Omega 03f5bb14c - Update MirahezeFunctions.php [03:04:19] miraheze/mw-config - Universal-Omega the build passed. [03:04:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.27, 3.54, 3.90 [03:05:31] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [03:05:33] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f5bb14c831a0...6681414bbcaf [03:05:34] [02miraheze/mw-config] 07Universal-Omega 036681414 - Update MirahezeFunctions.php [03:06:39] miraheze/mw-config - Universal-Omega the build passed. [03:08:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.45, 4.45, 4.14 [03:10:03] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [03:10:04] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6681414bbcaf...b7ba5fadf2b7 [03:10:06] [02miraheze/mw-config] 07Universal-Omega 03b7ba5fa - Update MirahezeFunctions.php [03:10:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.50, 3.30 [03:11:10] miraheze/mw-config - Universal-Omega the build passed. [03:12:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.29, 3.31, 3.24 [03:13:13] [02mw-config] 07Universal-Omega synchronize pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [03:13:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b7ba5fadf2b7...6b6da181ce02 [03:13:16] [02miraheze/mw-config] 07Universal-Omega 036b6da18 - Update LocalSettings.php [03:14:32] miraheze/mw-config - Universal-Omega the build passed. [03:15:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:16:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±9] 13https://github.com/miraheze/mw-config/compare/170e2b370f66...4a29c247f780 [03:16:21] [02miraheze/mw-config] 07Universal-Omega 034a29c24 - Rewrite SiteConfiguration setup (#4465) [03:16:22] [02mw-config] 07Universal-Omega closed pull request 03#4465: Rewrite SiteConfiguration setup - 13https://github.com/miraheze/mw-config/pull/4465 [03:16:24] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [03:16:25] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [03:16:54] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [03:16:56] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 2s [03:16:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:27] miraheze/mw-config - Universal-Omega the build passed. [03:17:35] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [03:17:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:18:34] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [03:18:41] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [03:18:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:18:52] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [03:19:09] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [03:19:20] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [03:19:29] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [03:19:38] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [03:19:46] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [03:20:10] RECOVERY - id.altilunium.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'id.altilunium.xyz' will expire on Mon 11 Jul 2022 01:10:38 GMT +0000. [03:20:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:25:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.70, 3.47, 3.25 [03:26:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.32, 10.18, 7.89 [03:27:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.06, 2.97, 3.09 [03:27:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:28:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.51, 11.30, 8.56 [03:28:32] !log [universalomega@mwtask111] starting deploy of {'config': True} to all [03:28:49] !log [universalomega@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 16s [03:28:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:30:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.22, 10.57, 8.63 [03:30:19] !log locally reverted 4a29c247f7803c7841c6602148c1ff3c0f6d9840 on mwtask111 config [03:30:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:31:30] PROBLEM - mwtask111 Puppet on mwtask111 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 9 minutes ago with 0 failures [03:32:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.44, 8.78, 8.21 [03:32:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:37:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4a29c247f780...6762e2ace48b [03:37:21] [02miraheze/mw-config] 07Universal-Omega 036762e2a - fix wgServer values [03:37:45] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [03:37:46] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [03:37:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:37:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:38:25] miraheze/mw-config - Universal-Omega the build passed. [03:41:09] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [03:41:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:41:20] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [03:41:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:42:55] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [03:42:56] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 1s [03:43:30] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:43:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:43:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:45:19] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:45:37] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [03:45:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:45:55] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 18s [03:46:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.59, 9.63, 8.33 [03:46:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:47:04] !log [universalomega@test101] starting deploy of {'config': True, 'ignoretime': True} to all [03:47:05] !log [universalomega@test101] finished deploy of {'config': True, 'ignoretime': True} to all - SUCCESS in 1s [03:47:16] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.830 second response time [03:47:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:47:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:48:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.70, 9.99, 8.63 [03:48:16] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.06, 10.63, 8.76 [03:48:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:49:25] !log [@test101] starting deploy of {'config': True} to all [03:49:26] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [03:49:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:49:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:50:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.27, 10.56, 8.98 [03:50:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.57, 3.40, 3.99 [03:51:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.93, 1.26, 0.63 [03:52:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.14, 10.45, 9.15 [03:53:03] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.30, 1.30, 0.73 [03:54:42] [02mw-config] 07Universal-Omega synchronize pull request 03#4539: Try setting parsoid url to localhost - 13https://github.com/miraheze/mw-config/pull/4539 [03:54:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f672b512c9a6...d84876624a54 [03:54:45] [02miraheze/mw-config] 07Universal-Omega 03d848766 - Update Defines.php [03:55:00] [02mw-config] 07Universal-Omega edited pull request 03#4539: Try setting parsoid url to server hostname directly - 13https://github.com/miraheze/mw-config/pull/4539 [03:55:51] miraheze/mw-config - Universal-Omega the build passed. [03:56:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.37, 11.09, 9.68 [03:56:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.43, 10.36, 8.87 [03:56:45] !log [universalomega@test101] starting deploy of {'config': True, 'ignoretime': True} to all [03:56:45] !log [universalomega@test101] finished deploy of {'config': True, 'ignoretime': True} to all - SUCCESS in 1s [03:56:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:56:58] [02mw-config] 07Universal-Omega closed pull request 03#4539: Try setting parsoid url to server hostname directly - 13https://github.com/miraheze/mw-config/pull/4539 [03:57:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6762e2ace48b...b3b12f3a71b2 [03:57:01] [02miraheze/mw-config] 07Universal-Omega 03b3b12f3 - Try setting parsoid url to server hostname directly (#4539) [03:57:03] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [03:57:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:57:04] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [03:57:25] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [03:57:27] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 1s [03:57:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:57:39] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [03:57:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:57:55] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 15s [03:57:56] miraheze/mw-config - Universal-Omega the build passed. [03:57:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:57:59] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.06, 9.48, 8.73 [03:58:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:58:44] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.40, 2.41, 1.81 [04:00:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.81, 11.24, 10.13 [04:00:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.56, 11.95, 10.71 [04:00:43] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.94, 1.88, 1.69 [04:01:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.27, 10.38, 9.30 [04:02:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.22, 11.53, 10.36 [04:02:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.90, 1.61, 1.62 [04:03:49] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.97, 9.47, 9.11 [04:03:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.70, 3.55, 3.59 [04:06:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.84, 11.89, 10.83 [04:06:56] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [04:06:57] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 1s [04:07:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:07:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:07:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.46, 3.55 [04:08:51] [02mediawiki] 07Universal-Omega closed pull request 03#6284: Bump extensions/ManageWiki from `740bb6e` to `9780e21` - 13https://github.com/miraheze/mediawiki/pull/6284 [04:08:52] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/c2364ade75df...a1e72c203970 [04:08:54] [02miraheze/mediawiki] 07dependabot[bot] 03a1e72c2 - Bump extensions/ManageWiki from `740bb6e` to `9780e21` (#6284) [04:08:55] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ManageWiki-9780e21 - 13https://github.com/miraheze/mediawiki [04:08:57] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ManageWiki-9780e21 [04:09:14] !log [universalomega@test101] starting deploy of {'world': True} to all [04:09:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:09:33] !log [universalomega@mwtask111] starting deploy of {'world': True} to all [04:09:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:09:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.61, 3.58 [04:10:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.48, 11.29, 10.74 [04:11:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.65, 3.60 [04:12:01] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.11, 11.00, 10.67 [04:12:37] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.58, 1.99, 1.71 [04:13:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.79, 4.13, 3.78 [04:14:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.70, 11.18, 10.77 [04:14:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.93, 11.02, 10.70 [04:14:15] !log [universalomega@test101] finished deploy of {'world': True} to all - SUCCESS in 301s [04:14:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:14:36] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.72, 1.55, 1.58 [04:15:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.56, 3.16 [04:15:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.27, 3.70, 3.67 [04:15:46] !log [universalomega@mwtask111] finished deploy of {'world': True} to all - SUCCESS in 373s [04:15:58] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.12, 11.79, 11.04 [04:16:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:17:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.30, 3.18, 3.08 [04:17:57] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.96, 11.63, 11.06 [04:19:55] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.15, 1.88, 1.49 [04:19:56] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.31, 12.24, 11.35 [04:21:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.31, 2.90, 3.32 [04:21:51] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.39, 1.68, 1.46 [04:21:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.82, 11.57, 11.20 [04:28:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.42, 10.98, 10.61 [04:30:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.67, 10.97, 10.66 [04:31:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.16, 3.41, 3.29 [04:32:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.80, 10.73, 9.91 [04:33:40] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.66, 10.60, 9.69 [04:33:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.09, 2.89, 3.11 [04:35:37] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.47, 11.43, 10.09 [04:35:51] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.87, 11.45, 10.43 [04:37:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.86, 9.41, 10.13 [04:41:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.66, 9.94, 10.22 [04:41:58] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [04:43:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.47, 9.83, 10.15 [04:44:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.53, 3.52, 2.93 [04:46:23] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.47, 3.06, 2.82 [04:47:16] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.48, 11.17, 10.93 [04:47:27] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.46, 11.51, 10.88 [04:47:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.46, 10.78, 10.48 [04:49:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.59, 11.06, 10.79 [04:49:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.88, 11.37, 10.72 [04:51:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.22, 1.77, 1.41 [04:51:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.10, 11.20, 10.74 [04:52:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.83, 10.71, 10.47 [04:53:03] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.65, 1.68, 1.42 [04:55:02] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.61, 10.47, 10.52 [04:55:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.10, 11.33, 10.88 [04:56:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.47, 11.19, 10.75 [04:56:59] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.85, 10.67, 10.59 [04:57:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.64, 11.33, 10.94 [04:58:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.45, 12.40, 11.25 [04:59:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.00, 11.72, 11.09 [05:01:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.75, 11.28, 10.95 [05:01:42] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.10, 11.35, 11.03 [05:02:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.01, 11.70, 11.27 [05:03:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/42b3ba7f6e55 [05:03:42] [02miraheze/mw-config] 07Universal-Omega 0342b3ba7 - Use mw-lb for parsoid URL [05:03:44] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [05:03:45] [02mw-config] 07Universal-Omega opened pull request 03#4564: Use mw-lb for parsoid URL - 13https://github.com/miraheze/mw-config/pull/4564 [05:04:14] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/42b3ba7f6e55...6b195a6f73c9 [05:04:15] [02miraheze/mw-config] 07github-actions 036b195a6 - CI: lint code to MediaWiki standards [05:04:17] [02mw-config] 07github-actions[bot] synchronize pull request 03#4564: Use mw-lb for parsoid URL - 13https://github.com/miraheze/mw-config/pull/4564 [05:04:53] miraheze/mw-config - Universal-Omega the build passed. [05:06:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b3b12f3a71b2...ba63134fff18 [05:06:02] [02miraheze/mw-config] 07Universal-Omega 03ba63134 - Fix formatting [05:07:05] miraheze/mw-config - Universal-Omega the build passed. [05:07:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.16, 11.47, 11.24 [05:07:36] [02miraheze/mw-config] 07Universal-Omega pushed 032 commits to 03Universal-Omega-patch-1 [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/6b195a6f73c9...61e671286edd [05:07:38] [02miraheze/mw-config] 07Universal-Omega 0361e6712 - Merge branch 'master' into Universal-Omega-patch-1 [05:07:39] [02mw-config] 07Universal-Omega synchronize pull request 03#4564: Use mw-lb for parsoid URL - 13https://github.com/miraheze/mw-config/pull/4564 [05:07:51] [02mw-config] 07Universal-Omega synchronize pull request 03#4564: Use mw-lb for parsoid URL - 13https://github.com/miraheze/mw-config/pull/4564 [05:07:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/61e671286edd...dd46b56d64b6 [05:07:54] [02miraheze/mw-config] 07Universal-Omega 03dd46b56 - Update MirahezeFunctions.php [05:08:49] miraheze/mw-config - Universal-Omega the build passed. [05:08:50] miraheze/mw-config - Universal-Omega the build passed. [05:09:17] [02mw-config] 07Universal-Omega closed pull request 03#4564: Use mw-lb for parsoid URL - 13https://github.com/miraheze/mw-config/pull/4564 [05:09:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ba63134fff18...853a16613cf0 [05:09:20] [02miraheze/mw-config] 07Universal-Omega 03853a166 - Use mw-lb for parsoid URL (#4564) [05:09:21] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [05:09:23] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [05:09:58] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:10:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.81, 3.77, 3.03 [05:10:18] miraheze/mw-config - Universal-Omega the build passed. [05:10:35] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.55, 11.02, 10.62 [05:12:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.45, 3.09, 2.86 [05:12:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.42, 10.51, 10.50 [05:13:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.20, 9.50, 10.15 [05:15:57] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.54, 3.89, 3.20 [05:17:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.64, 3.53, 3.16 [05:19:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.52, 1.34 [05:19:26] !log [@test101] starting deploy of {'config': True} to all [05:19:27] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [05:19:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:19:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.28, 10.84, 10.45 [05:19:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:19:54] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.66, 3.16, 3.07 [05:20:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.35, 11.11, 10.83 [05:20:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.10, 10.75, 10.54 [05:21:03] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.30, 1.48, 1.35 [05:21:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.00, 12.09, 11.37 [05:22:15] !log [@mwtask111] starting deploy of {'config': True} to all [05:22:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:22:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.03, 9.77, 10.21 [05:22:27] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 12s [05:22:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:23:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.75, 10.60, 10.47 [05:24:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.87, 11.19, 10.98 [05:24:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.69, 10.60, 10.44 [05:26:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.27, 10.97, 10.61 [05:28:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.01, 10.71, 10.51 [05:30:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.32, 10.12, 9.50 [05:30:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.76, 10.95, 10.62 [05:32:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.29, 10.00, 9.55 [05:32:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.09, 10.44, 9.59 [05:32:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.78, 11.56, 10.88 [05:34:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.52, 10.75, 9.81 [05:35:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.24, 9.93, 10.18 [05:36:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.65, 10.34, 9.82 [05:36:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.61, 11.09, 10.02 [05:36:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.77, 10.92, 10.80 [05:38:08] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.06, 9.91, 9.71 [05:42:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.21, 11.00, 10.20 [05:42:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.95, 9.11, 9.95 [05:43:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.51, 10.43, 10.09 [05:44:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.89, 10.11, 9.94 [05:45:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.58, 9.57, 9.82 [05:50:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.06, 11.34, 10.60 [05:52:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.78, 9.75, 9.78 [05:53:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.88, 10.88, 11.72 [05:54:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.59, 10.67, 10.52 [05:55:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.26, 11.19, 11.71 [05:56:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.80, 9.84, 9.84 [05:56:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.01, 11.33, 10.79 [05:57:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.66, 10.58, 11.40 [05:58:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.10, 9.60, 10.10 [05:58:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.41, 10.93, 10.70 [05:59:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.05, 10.90, 11.40 [06:00:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.24, 10.73, 10.26 [06:02:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.51, 9.76, 9.95 [06:06:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.32, 9.84, 9.43 [06:06:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.88, 10.16, 9.85 [06:07:35] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.39, 1.68, 1.38 [06:08:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.70, 10.22, 10.00 [06:08:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 10.08, 9.77, 9.45 [06:08:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.48, 9.58, 9.67 [06:08:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.83, 9.97, 9.91 [06:09:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.67, 11.41, 11.64 [06:10:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.36, 10.77, 10.21 [06:10:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.93, 9.97, 9.92 [06:12:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.04, 10.68, 10.25 [06:13:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.52, 11.52, 11.59 [06:14:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.65, 10.19, 10.11 [06:14:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.49, 10.33, 9.95 [06:15:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.00, 11.27, 11.49 [06:15:59] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.39, 9.97, 9.86 [06:16:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.91, 10.36, 10.14 [06:18:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.08, 9.49, 9.84 [06:20:15] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.11, 11.41, 10.82 [06:21:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.59, 11.79, 11.54 [06:23:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.95, 11.79, 11.58 [06:24:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.47, 11.12, 10.49 [06:25:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.43, 11.65, 11.53 [06:25:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.95, 10.58, 10.04 [06:26:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 12.00, 11.34, 10.64 [06:26:29] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.41, 10.35, 10.13 [06:27:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.73, 11.61, 11.53 [06:28:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.95, 9.90, 9.30 [06:28:10] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.15, 11.14, 11.01 [06:29:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.01, 12.11, 11.72 [06:29:33] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.03, 10.14, 10.01 [06:30:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.89, 9.61, 9.28 [06:32:08] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.45, 11.45, 11.13 [06:32:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.10, 10.95, 10.43 [06:33:52] PROBLEM - test101 APT on test101 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:34:06] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.15, 11.15, 11.08 [06:34:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.23, 10.19, 10.21 [06:36:01] RECOVERY - test101 APT on test101 is OK: APT OK: 56 packages available for upgrade (0 critical updates). [06:37:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.30, 11.75, 11.89 [06:38:04] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.13, 11.74, 11.34 [06:40:03] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.34, 10.80, 11.05 [06:41:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.54, 12.40, 12.08 [06:42:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.06, 10.21, 9.99 [06:42:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.98, 11.45, 10.75 [06:43:48] PROBLEM - test101 APT on test101 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [06:45:57] RECOVERY - test101 APT on test101 is OK: APT OK: 56 packages available for upgrade (0 critical updates). [06:47:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.60, 11.69, 11.86 [06:48:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.20, 11.58, 11.11 [06:49:56] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.08, 11.29, 11.02 [06:50:04] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.16, 9.65, 9.93 [06:52:13] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [06:53:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.05, 11.42, 11.14 [06:56:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.24, 11.68, 11.13 [06:57:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.49, 11.50, 11.55 [06:59:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.07, 10.74, 11.26 [07:01:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.42, 11.45, 11.46 [07:03:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.63, 11.18, 11.34 [07:06:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.51, 9.37, 10.18 [07:07:45] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.01, 9.33, 10.17 [07:10:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.07, 11.92, 11.89 [07:10:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.33, 10.55, 10.44 [07:11:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.74, 9.91, 9.40 [07:12:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.55, 9.72, 10.15 [07:13:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.92, 9.78, 9.41 [07:13:31] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:14:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.39, 12.22, 12.04 [07:16:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.89, 11.38, 11.76 [07:20:12] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:23:31] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:24:37] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.89, 10.81, 10.33 [07:25:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.51, 11.33, 10.86 [07:26:08] Reception123: delbackups/deletions [07:27:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.57, 10.68, 10.22 [07:29:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.79, 11.72, 11.15 [07:29:14] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.17, 11.49, 10.56 [07:30:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.54, 10.10, 10.14 [07:31:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.76, 11.95, 11.29 [07:33:08] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.83, 11.01, 10.63 [07:37:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.63, 11.45, 11.37 [07:40:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.00, 9.61, 10.20 [07:40:54] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.94, 11.14, 10.64 [07:42:50] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.02, 10.92, 10.61 [07:43:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:48:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.57, 10.43, 9.83 [07:48:40] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.66, 9.65, 10.12 [07:51:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:52:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.27, 10.14, 9.97 [07:52:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.85, 11.43, 10.44 [07:54:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.40, 9.70, 9.85 [07:55:15] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.41, 11.48, 11.30 [07:56:29] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.43, 10.68, 10.39 [07:57:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.84, 11.61, 11.37 [07:59:13] .op [07:59:13] Attempting to OP... [08:00:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.35, 11.38, 10.71 [08:01:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.40, 11.58, 11.40 [08:02:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.95, 10.93, 10.65 [08:04:11] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.23, 9.85, 10.12 [08:05:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.27, 11.75, 11.57 [08:06:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:10:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.58, 11.51, 10.99 [08:12:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.46, 10.54, 9.86 [08:12:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.69, 9.91, 9.34 [08:12:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.70, 11.78, 11.17 [08:12:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:14:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.26, 9.73, 9.35 [08:17:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.74, 11.91, 11.39 [08:19:45] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.96, 9.44, 9.69 [08:22:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.91, 10.47, 9.81 [08:23:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.92, 11.35, 11.45 [08:24:07] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.42, 10.81, 9.99 [08:26:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.71, 10.66, 10.04 [08:27:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.90, 11.63, 11.48 [08:28:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.14, 10.16, 9.93 [08:30:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.42, 8.98, 9.91 [08:33:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.16, 11.81, 11.70 [08:36:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.67, 10.09, 9.58 [08:38:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.73, 9.37, 9.36 [08:39:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.32, 10.20, 10.09 [08:43:12] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.19, 9.96, 10.04 [08:50:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.23, 9.88, 9.52 [08:52:54] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.92, 9.30, 9.37 [08:55:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.06, 1.42, 1.89 [08:55:57] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.21, 11.21, 10.40 [08:57:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.92, 11.07, 10.45 [08:59:03] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.00, 1.20, 1.68 [08:59:50] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.97, 11.73, 10.76 [09:01:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.59, 11.73, 10.98 [09:01:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.33, 11.42, 10.75 [09:03:43] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.03, 11.43, 10.83 [09:05:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.15, 11.31, 11.07 [09:05:40] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.06, 10.74, 10.63 [09:07:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.94, 10.48, 9.93 [09:09:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.11, 11.49, 11.16 [09:09:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.51, 9.85, 9.78 [09:11:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.43, 11.36, 11.14 [09:11:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.51, 10.55, 9.76 [09:11:29] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.04, 11.29, 10.91 [09:15:22] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.55, 10.71, 10.77 [09:15:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.62, 10.15, 9.79 [09:17:19] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.00, 11.46, 11.03 [09:19:15] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.35, 10.85, 10.85 [09:25:13] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.80, 10.92, 9.97 [09:27:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.79, 10.95, 10.08 [09:28:58] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.71, 9.41, 10.10 [09:31:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.84, 10.67, 10.59 [09:32:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:33:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.66, 9.63, 10.23 [09:35:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.17, 9.49, 9.85 [09:35:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.00, 9.27, 10.04 [09:39:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:54:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:59:43] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.45, 10.81, 9.74 [10:01:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.93, 11.08, 9.96 [10:05:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.55, 10.09, 9.80 [10:09:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.43, 10.33, 9.98 [10:11:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.55, 9.72, 8.84 [10:12:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.41, 10.57, 9.84 [10:13:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.90, 9.16, 8.76 [10:14:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:15:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.76, 10.18, 10.03 [10:19:50] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 10.00, 10.07, 9.91 [10:24:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:27:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.03, 9.91, 9.31 [10:29:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:29:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.91, 9.22, 9.13 [10:39:14] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [10:46:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.76, 10.26, 9.60 [10:46:30] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.28, 10.30, 9.46 [10:48:28] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.36, 11.35, 9.95 [10:50:09] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.19, 11.07, 10.05 [10:51:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 1.88, 2.05, 1.56 [10:52:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.91, 11.50, 10.33 [10:52:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.89, 11.42, 10.29 [10:53:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.29, 1.77, 1.51 [10:55:03] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.18, 1.54, 1.46 [10:55:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.42, 11.37, 10.14 [10:57:19] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.98, 11.44, 10.20 [10:57:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.57, 11.41, 10.31 [10:59:15] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.25, 10.57, 10.03 [11:00:21] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.35, 11.71, 10.84 [11:01:22] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.80, 10.14, 10.12 [11:02:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.49, 4.60, 3.27 [11:02:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.29, 10.85, 10.61 [11:03:42] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.47, 12.20, 11.11 [11:04:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.90, 3.92, 3.18 [11:05:17] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.04, 11.29, 10.53 [11:06:17] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.64, 11.51, 10.90 [11:07:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.53, 10.71, 10.42 [11:08:39] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [11:09:59] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.86, 2.90, 3.02 [11:10:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.52, 11.47, 11.02 [11:10:55] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.89, 11.26, 10.55 [11:11:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.44, 9.78, 10.09 [11:11:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.58, 11.93, 11.47 [11:12:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.62, 10.51, 9.62 [11:13:21] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.69, 12.17, 11.61 [11:15:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.66, 11.92, 11.60 [11:16:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.06, 11.73, 11.11 [11:18:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.87, 12.00, 11.30 [11:18:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.10, 9.89, 9.72 [11:20:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.90, 11.89, 11.35 [11:21:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.74, 11.70, 11.57 [11:21:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.66, 11.27, 10.44 [11:23:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.76, 11.63, 11.56 [11:23:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.88, 11.22, 10.51 [11:24:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.07, 11.85, 11.43 [11:25:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.04, 11.61, 10.72 [11:26:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.95, 11.78, 11.48 [11:26:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.44, 11.89, 11.38 [11:28:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.34, 10.93, 11.09 [11:30:02] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.13, 11.60, 11.43 [11:31:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.06, 11.60, 11.42 [11:31:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.30, 10.98, 10.82 [11:32:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.30, 10.23, 10.94 [11:32:50] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:32:58] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:33:00] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:33:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.51, 11.73, 11.51 [11:34:49] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.886 second response time [11:34:58] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 3.439 second response time [11:35:03] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.776 second response time [11:35:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.97, 12.22, 11.72 [11:35:24] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.21, 11.04, 10.81 [11:36:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.70, 9.81, 9.43 [11:37:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.09, 11.36, 11.47 [11:38:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.74, 11.16, 10.97 [11:38:48] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.46, 9.56, 9.37 [11:39:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.98, 11.59, 11.09 [11:40:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.58, 10.68, 10.81 [11:41:54] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.21, 12.00, 11.49 [11:42:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.64, 11.52, 11.11 [11:43:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.05, 11.58, 11.48 [11:43:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.98, 11.23, 11.27 [11:44:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.00, 11.48, 11.15 [11:46:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.41, 12.18, 11.45 [11:47:33] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.36, 10.24, 9.67 [11:48:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.21, 11.79, 11.39 [11:48:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.38, 12.08, 11.39 [11:49:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.64, 1.68, 1.35 [11:49:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.06, 9.22, 9.38 [11:50:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.28, 11.94, 11.49 [11:51:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.16, 2.91 [11:52:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.26, 11.96, 11.51 [11:53:03] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.31, 1.61, 1.42 [11:53:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.25, 3.21, 2.96 [11:54:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.47, 10.50, 9.66 [11:58:46] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.54, 10.00, 9.66 [12:01:20] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:59] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:01] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:18] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:04:04] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:05:42] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.74, 9.01, 10.11 [12:06:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.88, 10.36, 9.96 [12:08:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.35, 9.96, 9.85 [12:08:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.83, 11.39, 11.16 [12:08:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.77, 11.46, 11.80 [12:09:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.57, 10.11, 9.54 [12:11:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.89, 11.83, 11.95 [12:11:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.88, 9.92, 9.54 [12:13:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.96, 12.15, 12.05 [12:13:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.59, 10.06, 10.15 [12:14:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.28, 11.81, 11.47 [12:14:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.84, 11.18, 11.45 [12:15:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.47, 10.10, 10.17 [12:16:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.99, 11.39, 11.50 [12:18:03] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.815 second response time [12:18:05] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.145 second response time [12:18:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.09, 12.09, 11.75 [12:18:44] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.107 second response time [12:18:45] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.909 second response time [12:19:05] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.793 second response time [12:20:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.39, 11.80, 11.69 [12:21:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.68, 11.74, 11.96 [12:21:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.38, 11.11, 10.52 [12:22:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.49, 12.05, 11.54 [12:22:07] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:07] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:39] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:15] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.56, 10.97, 10.54 [12:24:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.50, 11.78, 11.50 [12:25:20] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.255 second response time [12:25:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.71, 11.67, 10.85 [12:26:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.03, 12.06, 11.65 [12:26:21] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.016 second response time [12:26:23] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 8.276 second response time [12:26:48] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.581 second response time [12:27:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.85, 11.17, 10.76 [12:30:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.12, 11.11, 11.20 [12:32:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.94, 11.82, 11.72 [12:32:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.95, 10.79, 11.06 [12:32:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.30, 10.37, 9.90 [12:33:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.10, 12.20, 11.83 [12:33:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.29, 11.25, 10.87 [12:35:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.25, 10.50, 10.65 [12:36:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.01, 11.79, 11.68 [12:36:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.95, 9.89, 9.29 [12:36:51] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.37, 10.13, 9.94 [12:37:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.85, 11.76, 11.78 [12:38:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.32, 11.09, 11.43 [12:39:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.57, 12.14, 11.93 [12:39:45] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [12:39:45] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:40:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.91, 9.73, 9.38 [12:40:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.69, 12.09, 11.49 [12:40:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.93, 11.15, 10.39 [12:41:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.86, 11.35, 11.66 [12:45:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.42, 11.47, 11.60 [12:46:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.28, 11.76, 11.57 [12:48:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.05, 11.71, 11.56 [12:48:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.35, 10.94, 10.10 [12:49:14] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.36, 1.68, 1.34 [12:49:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.47, 11.03, 10.68 [12:50:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.64, 11.77, 11.61 [12:50:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.32, 9.44, 9.66 [12:51:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.39, 1.57, 1.35 [12:51:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.80, 11.08, 10.76 [12:52:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.54, 11.90, 11.53 [12:52:21] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.54, 9.90, 10.15 [12:53:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.74, 11.68, 11.80 [12:54:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.92, 11.80, 11.56 [12:55:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.34, 11.68, 11.77 [12:56:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.00, 12.09, 11.71 [12:56:08] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:53] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 3.64, 3.16 [12:57:32] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:15] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.573 second response time [12:58:51] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.22, 3.18, 3.05 [12:59:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.03, 11.69, 11.04 [13:01:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.17, 11.11, 10.91 [13:02:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.64, 12.00, 11.84 [13:02:35] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 1.529 second response time [13:02:56] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 1.799 second response time [13:03:07] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:10] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.31, 11.64, 11.82 [13:04:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.19, 12.10, 11.90 [13:09:10] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [13:09:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.54, 11.96, 11.87 [13:10:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.34, 11.97, 11.97 [13:12:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.97, 11.32, 10.53 [13:13:23] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.371 second response time [13:13:44] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:45] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:14:20] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 7.161 second response time [13:14:48] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.79, 11.56, 11.75 [13:15:38] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.270 second response time [13:15:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.40, 9.26, 9.97 [13:15:44] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.955 second response time [13:15:45] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 5.230 second response time [13:16:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.50, 10.13, 9.57 [13:16:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.20, 11.18, 10.68 [13:17:15] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:18:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.62, 9.83, 9.55 [13:18:12] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:18:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.41, 11.50, 11.29 [13:19:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.11, 11.75, 11.78 [13:19:21] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.490 second response time [13:19:52] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 7.105 second response time [13:20:01] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.998 second response time [13:20:13] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.953 second response time [13:20:30] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.25, 11.51, 10.91 [13:20:58] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.022 second response time [13:21:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.11, 10.75, 10.34 [13:22:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.14, 11.03, 10.80 [13:25:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.56, 11.82, 11.87 [13:25:23] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.49, 10.91, 10.55 [13:26:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.20, 10.24, 9.74 [13:26:16] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:26:17] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:26:18] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:27:02] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.38, 9.81, 9.66 [13:30:10] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.74, 9.52, 10.19 [13:30:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.13, 11.45, 11.56 [13:31:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.12, 11.80, 11.84 [13:31:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.51, 11.33, 10.77 [13:33:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.64, 11.28, 10.82 [13:38:34] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [13:41:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.64, 9.35, 10.11 [13:42:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.98, 11.20, 10.89 [13:42:32] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:43:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.27, 10.01, 9.94 [13:44:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.97, 8.81, 9.87 [13:44:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.63, 11.08, 10.87 [13:44:32] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 5.480 second response time [13:45:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.71, 11.55, 11.90 [13:45:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.85, 10.16, 10.02 [13:46:11] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:08] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.360 second response time [13:48:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.80, 3.24 [13:49:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.79, 10.64, 10.29 [13:52:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.90, 3.04, 3.08 [13:54:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.12, 10.85, 10.03 [13:54:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.23, 9.81, 9.34 [13:55:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.39, 11.88, 11.78 [13:56:09] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 10.05, 9.79, 9.37 [13:57:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.37, 11.57, 11.67 [14:00:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.47, 10.82, 10.41 [14:01:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.12, 11.95, 11.78 [14:01:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.33, 9.69, 10.09 [14:04:01] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:04:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.60, 10.47, 9.90 [14:06:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.98, 9.47, 9.59 [14:10:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.61, 9.88, 10.16 [14:11:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.38, 11.41, 11.72 [14:11:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.69, 10.25, 9.70 [14:12:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.42, 10.81, 10.14 [14:12:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.76, 11.86, 11.17 [14:13:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.50, 11.76, 11.80 [14:13:45] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.20, 9.75, 9.57 [14:14:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.56, 10.12, 9.98 [14:14:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.75, 11.74, 11.21 [14:15:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.46, 11.66, 11.77 [14:15:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.89, 3.82, 3.06 [14:17:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.06, 11.84, 11.82 [14:17:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.64, 3.11 [14:19:01] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:19:31] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.455 second response time [14:19:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.81, 10.37, 9.96 [14:19:42] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 8.023 second response time [14:19:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.55, 3.21, 3.01 [14:20:57] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.230 second response time [14:21:05] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.299 second response time [14:21:22] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.816 second response time [14:21:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.46, 10.87, 10.18 [14:22:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.85, 11.04, 10.39 [14:22:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.43, 3.02 [14:23:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.76, 11.05, 10.33 [14:23:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.93, 3.38 [14:24:08] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:24:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.36, 2.97, 2.89 [14:25:22] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:30] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.25, 3.37, 3.24 [14:25:51] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:26:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.03, 10.08, 10.14 [14:28:00] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 9.797 second response time [14:28:18] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:30:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.53, 10.21, 10.21 [14:31:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.76, 10.09, 10.20 [14:31:56] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.86, 9.27, 9.86 [14:32:09] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:18] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:29] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:32] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:42] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.65, 3.35, 3.04 [14:33:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.54, 11.07, 11.72 [14:34:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.16, 3.86, 3.28 [14:36:20] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.900 second response time [14:36:31] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.038 second response time [14:36:37] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 6.816 second response time [14:36:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.63, 3.79, 3.33 [14:38:12] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.345 second response time [14:38:59] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.398 second response time [14:39:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 3.04, 2.72 [14:39:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.68, 11.30, 11.53 [14:39:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.43, 10.68, 10.38 [14:40:17] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [14:40:17] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:40:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.33, 9.83, 9.82 [14:40:58] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.87, 2.99, 2.74 [14:41:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.67, 10.97, 11.38 [14:41:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.72, 9.45, 9.95 [14:42:49] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:42:55] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:44:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.34, 3.87, 3.45 [14:44:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.05, 9.80, 9.79 [14:44:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.63, 3.30, 3.31 [14:46:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.93, 3.54, 3.38 [14:47:02] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.409 second response time [14:47:03] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 8.904 second response time [14:48:20] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.77, 3.48 [14:48:59] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:49:14] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.56, 1.81, 1.45 [14:50:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.67, 3.49 [14:50:33] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:50:43] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:51:11] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.30, 1.62, 1.42 [14:51:24] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:51:29] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.31, 11.09, 11.08 [14:54:15] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.50, 3.20, 3.35 [14:55:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.18, 10.49, 10.87 [14:58:01] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:58:30] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:03] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.935 second response time [14:59:08] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.966 second response time [14:59:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.27, 11.12, 11.02 [14:59:32] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.763 second response time [14:59:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.67, 9.83, 9.66 [14:59:42] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 9.490 second response time [14:59:54] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.388 second response time [15:00:02] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 6.164 second response time [15:00:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.33, 10.37, 10.14 [15:00:28] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.217 second response time [15:01:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.85, 11.00, 10.99 [15:01:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.71, 9.71, 9.64 [15:08:11] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.58, 10.04, 10.09 [15:09:15] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.12, 11.45, 11.17 [15:09:42] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [15:10:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.07, 11.40, 11.02 [15:11:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.77, 10.35, 9.37 [15:12:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.88, 11.45, 11.09 [15:15:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.87, 11.54, 11.29 [15:15:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.62, 3.79, 3.05 [15:17:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.03, 3.41, 3.00 [15:18:28] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.26, 10.73, 10.27 [15:19:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.80, 1.82, 1.43 [15:20:27] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.02, 11.55, 10.62 [15:21:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.68, 1.80, 1.47 [15:21:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.53, 3.11 [15:22:12] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [15:22:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.14, 11.28, 10.63 [15:23:03] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.23, 1.59, 1.43 [15:24:07] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:24:11] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:24:33] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:24:46] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:24:57] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:25:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.52, 9.92, 9.82 [15:25:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.74, 3.33, 3.13 [15:26:07] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.459 second response time [15:26:09] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.715 second response time [15:26:31] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.672 second response time [15:26:39] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:26:43] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 2.556 second response time [15:26:58] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.383 second response time [15:27:55] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.37, 11.38, 11.16 [15:29:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.07, 11.72, 11.37 [15:30:12] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:34] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:31:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.10, 11.39, 11.29 [15:32:18] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:32:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.57, 9.44, 10.11 [15:32:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.94, 11.38, 11.28 [15:36:16] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 8.977 second response time [15:36:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.17, 11.30, 11.22 [15:36:29] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.758 second response time [15:36:32] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.334 second response time [15:36:51] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.703 second response time [15:37:16] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.945 second response time [15:38:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.90, 11.05, 11.13 [15:40:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.14, 10.46, 10.00 [15:40:35] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:40:51] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:40:57] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:41:43] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:42:12] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.77, 10.08, 9.91 [15:43:01] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:08] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:20] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:22] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:24] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:37] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:44:18] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:04] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.459 second response time [15:45:16] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 9.263 second response time [15:45:21] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.678 second response time [15:45:32] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.661 second response time [15:46:18] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.137 second response time [15:47:41] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.019 second response time [15:48:50] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 7.319 second response time [15:49:15] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.402 second response time [15:49:23] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.244 second response time [15:49:34] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.616 second response time [15:50:04] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.207 second response time [15:50:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.78, 9.73, 9.57 [15:52:12] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:52:15] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 2.872 second response time [15:52:25] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 1.980 second response time [15:53:16] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.710 second response time [15:53:23] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.240 second response time [15:53:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.11, 3.50, 3.24 [15:53:34] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.243 second response time [15:54:05] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.25, 9.86, 9.68 [15:55:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.77, 10.61, 10.09 [15:55:30] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.19, 3.34, 3.20 [15:56:13] PROBLEM - puppet111 Current Load on puppet111 is CRITICAL: CRITICAL - load average: 11.86, 5.46, 2.68 [15:56:29] PROBLEM - puppet111 APT on puppet111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [15:57:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.37, 10.06, 9.95 [15:58:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.82, 11.62, 10.98 [15:58:29] RECOVERY - puppet111 APT on puppet111 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [15:59:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.38, 11.24, 10.85 [15:59:23] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:59:30] PROBLEM - mon111 Puppet on mon111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:59:40] PROBLEM - graylog121 Puppet on graylog121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:59:55] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:59:56] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:00:00] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.75, 11.12, 10.23 [16:00:10] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/home/macfan] [16:00:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.73, 11.38, 10.98 [16:00:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.57, 10.40, 9.99 [16:00:46] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:49] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.77, 10.25, 10.07 [16:01:09] PROBLEM - gluster121 Puppet on gluster121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:01:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.88, 11.07, 10.84 [16:01:22] PROBLEM - gluster111 Puppet on gluster111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:01:44] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:47] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.45, 10.40, 10.07 [16:01:59] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:02:11] RECOVERY - puppet111 Current Load on puppet111 is OK: OK - load average: 1.79, 4.88, 3.68 [16:02:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.71, 9.84, 9.85 [16:05:24] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:05:56] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.53, 9.90, 9.97 [16:06:20] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:06:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.54, 9.44, 9.85 [16:07:27] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 8.314 second response time [16:08:20] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.062 second response time [16:10:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.37, 9.02, 10.04 [16:13:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.68, 11.22, 10.94 [16:15:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.10, 11.16, 10.96 [16:16:36] [02mw-config] 07lens0021 opened pull request 03#4565: test: Validate ManageWikiSettings.php file - 13https://github.com/miraheze/mw-config/pull/4565 [16:17:46] miraheze/mw-config - lens0021 the build passed. [16:20:15] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:20:27] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:20:34] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:20:45] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:21:24] [02mw-config] 07lens0021 synchronize pull request 03#4565: test: Validate ManageWikiSettings.php file - 13https://github.com/miraheze/mw-config/pull/4565 [16:22:14] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.264 second response time [16:22:24] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.017 second response time [16:22:31] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.017 second response time [16:22:32] miraheze/mw-config - lens0021 the build passed. [16:22:43] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.340 second response time [16:23:57] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:24:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.24, 10.08, 9.82 [16:25:23] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [16:25:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.51, 10.95, 10.38 [16:25:55] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [16:27:22] RECOVERY - gluster111 Puppet on gluster111 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [16:27:40] RECOVERY - graylog121 Puppet on graylog121 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [16:27:56] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:28:39] [discord] meta is slow as hell [16:29:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.57, 10.20, 10.14 [16:29:09] RECOVERY - gluster121 Puppet on gluster121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:29:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.24, 11.54, 11.17 [16:29:30] RECOVERY - mon111 Puppet on mon111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:29:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.19, 9.99, 10.13 [16:30:52] [02mw-config] 07lens0021 edited pull request 03#4565: test: Validate ManageWikiSettings.php file - 13https://github.com/miraheze/mw-config/pull/4565 [16:31:03] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.03, 10.99, 10.45 [16:31:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.24, 11.39, 10.33 [16:32:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.54, 9.91, 9.93 [16:33:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.56, 10.66, 10.20 [16:34:56] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.69, 10.91, 10.56 [16:35:16] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.706 second response time [16:35:23] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.791 second response time [16:35:31] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.324 second response time [16:35:44] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.995 second response time [16:36:07] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.67, 11.12, 9.98 [16:36:34] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 4.603 second response time [16:36:53] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.37, 11.66, 10.86 [16:38:49] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.41, 11.33, 10.85 [16:39:01] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:39:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.20, 11.85, 10.83 [16:40:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.53, 10.72, 10.33 [16:40:46] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.52, 11.85, 11.10 [16:42:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.25, 10.45, 10.11 [16:42:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.36, 11.57, 10.58 [16:42:43] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [16:42:43] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:44:01] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:44:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.85, 11.63, 11.16 [16:44:39] [02mw-config] 07lens0021 synchronize pull request 03#4565: test: Validate ManageWikiSettings.php file - 13https://github.com/miraheze/mw-config/pull/4565 [16:45:38] miraheze/mw-config - lens0021 the build passed. [16:46:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.11, 10.12, 10.04 [16:46:36] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.62, 11.99, 11.35 [16:47:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.36, 11.96, 11.38 [16:48:01] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:17] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.58, 11.27, 10.66 [16:48:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.67, 11.58, 11.28 [16:49:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.10, 11.85, 11.40 [16:50:03] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.067 second response time [16:50:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.60, 9.39, 10.02 [16:50:22] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 8.236 second response time [16:51:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.70, 11.75, 11.42 [16:53:22] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:29] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:53] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.02, 11.29, 10.98 [16:54:29] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:48] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:56:55] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 5.32, 4.13, 2.08 [16:59:48] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.020 second response time [17:00:07] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.313 second response time [17:00:39] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.096 second response time [17:00:58] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 1.508 second response time [17:01:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.07, 2.73 [17:01:44] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.823 second response time [17:02:55] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.49, 1.56, 1.55 [17:03:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.03, 11.25, 11.15 [17:03:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.49, 3.87, 3.05 [17:04:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.03, 10.93, 10.39 [17:05:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.02, 10.62, 10.93 [17:05:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.03, 3.45, 3.00 [17:06:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.02, 10.71, 10.16 [17:06:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.47, 10.69, 10.37 [17:07:23] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:07:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.69, 4.00, 3.26 [17:08:11] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:08:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.01, 11.25, 11.10 [17:08:27] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:09:04] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:09:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.74, 3.25 [17:10:14] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.484 second response time [17:10:34] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.813 second response time [17:10:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.20, 9.82, 10.07 [17:11:06] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.679 second response time [17:11:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.40, 11.52, 11.95 [17:11:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.64, 11.17, 11.01 [17:11:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.75, 3.15, 3.10 [17:11:29] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 2.979 second response time [17:12:01] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:05] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:08] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [17:12:11] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:18] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:40] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.48, 10.61, 10.81 [17:14:07] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.768 second response time [17:14:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.95, 3.30, 2.75 [17:14:11] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.401 second response time [17:14:17] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.576 second response time [17:14:21] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 8.275 second response time [17:14:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.95, 11.59, 11.35 [17:14:44] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 5.592 second response time [17:15:35] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 670 bytes in 0.282 second response time [17:16:09] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.92, 3.00, 2.70 [17:16:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.26, 11.09, 10.89 [17:17:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.08, 11.46, 11.11 [17:17:38] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 7.305 second response time [17:17:44] [02mw-config] 07lens0021 synchronize pull request 03#4565: test: Validate ManageWikiSettings.php file - 13https://github.com/miraheze/mw-config/pull/4565 [17:18:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.12, 10.64, 10.28 [17:19:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.43, 11.35, 11.11 [17:20:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.69, 11.42, 11.11 [17:21:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.64, 1.72, 1.48 [17:21:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.65, 10.55, 10.94 [17:21:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.02, 11.80, 11.29 [17:21:35] [02mw-config] 07lens0021 edited pull request 03#4565: test: Validate ManageWikiSettings.php file - 13https://github.com/miraheze/mw-config/pull/4565 [17:22:04] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.09, 9.63, 9.98 [17:22:24] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:29] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:40] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:03] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.35, 1.55, 1.44 [17:23:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.13, 10.83, 11.00 [17:23:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.79, 10.95, 11.04 [17:24:00] miraheze/mw-config - lens0021 the build passed. [17:24:07] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.78, 11.50, 11.22 [17:27:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.41, 3.16, 3.02 [17:28:04] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.49, 9.26, 10.16 [17:28:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.96, 10.67, 10.24 [17:29:29] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.23, 3.46, 3.14 [17:30:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.12, 11.42, 11.25 [17:31:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.58, 3.23 [17:32:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.72, 11.34, 11.24 [17:33:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.11, 3.04, 3.07 [17:34:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.61, 9.19, 10.04 [17:35:30] [02mw-config] 07lens0021 synchronize pull request 03#4565: test: Validate ManageWikiSettings.php file - 13https://github.com/miraheze/mw-config/pull/4565 [17:36:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.71, 9.46, 9.94 [17:37:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.47, 10.66, 10.61 [17:37:51] miraheze/mw-config - lens0021 the build passed. [17:38:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.30, 11.56, 11.33 [17:39:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.99, 10.22, 10.44 [17:41:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.07, 3.48, 3.23 [17:42:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.50, 11.85, 11.51 [17:43:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.20, 3.04, 3.10 [17:44:27] [02mw-config] 07lens0021 synchronize pull request 03#4565: test: Validate ManageWikiSettings.php file - 13https://github.com/miraheze/mw-config/pull/4565 [17:45:36] miraheze/mw-config - lens0021 the build passed. [17:46:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.42, 10.23, 10.01 [17:46:50] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.00, 10.26, 9.89 [17:47:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.17, 10.93, 10.60 [17:48:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.55, 10.06, 9.97 [17:48:49] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.56, 11.08, 10.23 [17:49:22] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.538 second response time [17:49:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.20, 10.54, 10.50 [17:50:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.53, 10.34, 9.92 [17:50:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 12.00, 11.38, 10.45 [17:51:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.32, 11.83, 11.97 [17:52:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.90, 11.17, 10.27 [17:53:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.43, 9.63, 10.11 [17:53:37] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 5.824 second response time [17:54:06] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.021 second response time [17:54:13] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.229 second response time [17:54:42] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [17:55:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.28, 1.82, 1.50 [17:56:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.69, 11.08, 10.38 [17:57:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/853a16613cf0...2ba48b038f1e [17:57:07] [02miraheze/mw-config] 07Universal-Omega 032ba48b0 - T9064: fix `$wgConf->settings` for `$wgConf` settings set in MirahezeFunctions [17:57:38] !log [universalomega@mwtask111] starting deploy of {'config': True, 'ignoretime': True} to all [17:57:41] !log [universalomega@test101] starting deploy of {'config': True, 'ignoretime': True} to all [17:57:42] !log [universalomega@test101] finished deploy of {'config': True, 'ignoretime': True} to all - SUCCESS in 0s [17:57:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:58:05] !log [universalomega@mwtask111] finished deploy of {'config': True, 'ignoretime': True} to all - SUCCESS in 27s [17:58:14] miraheze/mw-config - Universal-Omega the build passed. [17:58:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:58:43] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.42, 11.24, 10.69 [17:58:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:58:55] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [17:59:04] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [17:59:06] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 1s [17:59:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:59:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.23, 12.38, 12.10 [17:59:17] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 22s [17:59:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:59:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.19, 10.79, 10.47 [17:59:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:59:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:59:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:00:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.68, 11.86, 11.10 [18:00:42] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.91, 10.48, 10.50 [18:01:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.73, 1.93, 1.66 [18:02:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.76, 9.98, 10.15 [18:04:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.80, 9.09, 9.98 [18:05:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.25, 9.18, 9.89 [18:06:38] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.43, 9.51, 10.07 [18:07:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.50, 2.39, 1.89 [18:07:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.19, 11.09, 11.78 [18:08:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.43, 9.21, 10.11 [18:10:02] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2ba48b038f1e...544491756e05 [18:10:03] [02miraheze/mw-config] 07Universal-Omega 035444917 - Fix `$wgCentralAuthLoginIcon` to point to new path [18:10:24] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [18:10:32] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [18:10:34] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [18:10:46] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 21s [18:10:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:11:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.36, 1.82, 1.78 [18:11:09] miraheze/mw-config - Universal-Omega the build passed. [18:11:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.32, 10.52, 10.33 [18:11:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:11:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:12:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.04, 10.79, 10.36 [18:12:55] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [18:13:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.18, 2.00, 1.85 [18:14:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.79, 10.69, 10.38 [18:15:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.71, 3.21 [18:15:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.06, 10.73, 11.18 [18:16:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.58, 11.05, 10.49 [18:17:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.32, 3.66, 3.26 [18:18:10] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.30, 10.68, 10.26 [18:18:57] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.98, 9.70, 10.03 [18:19:05] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.08, 3.39, 3.20 [18:20:26] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.59, 9.88, 10.14 [18:21:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.68, 2.00, 1.94 [18:21:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.62, 11.76, 11.54 [18:22:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.53, 9.99, 10.08 [18:22:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.17, 11.23, 10.73 [18:23:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.57, 2.03, 1.94 [18:24:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.34, 10.79, 10.62 [18:24:50] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.28, 11.49, 10.59 [18:25:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.30, 1.77, 1.86 [18:26:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.63, 10.68, 10.41 [18:26:47] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.72, 11.52, 10.72 [18:27:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.34, 11.26, 11.31 [18:27:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/544491756e05...93a761af6bfb [18:27:24] [02miraheze/mw-config] 07Universal-Omega 0393a761a - move `defined( 'MEDIAWIKI' )` check to top [18:27:36] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [18:27:37] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 2s [18:27:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:27:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:28:08] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [18:28:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:28:25] !log sudo rm -rf /usr/share/nginx/favicons on mw*, mwtask111, and test101 [18:28:26] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 18s [18:28:30] miraheze/mw-config - Universal-Omega the build passed. [18:28:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:29:03] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.00, 1.36, 1.67 [18:29:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:29:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.85, 11.24, 11.29 [18:29:53] !log [universalomega@test101] starting deploy of {'l10n': True} to all [18:30:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:30:10] !log [universalomega@mwtask111] starting deploy of {'l10n': True} to all [18:30:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:31:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.29, 12.02, 11.57 [18:33:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.34, 1.81, 1.77 [18:33:52] PROBLEM - test101 APT on test101 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [18:34:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.20, 11.19, 10.73 [18:35:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.58, 11.91, 11.65 [18:36:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.31, 11.02, 10.72 [18:37:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.34, 3.18 [18:39:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.06, 3.26, 3.17 [18:40:21] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.33, 11.68, 10.98 [18:40:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.45, 11.67, 10.99 [18:41:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.39, 10.01, 9.65 [18:42:11] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.18, 11.46, 10.90 [18:42:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.54, 11.38, 10.96 [18:42:19] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [18:42:19] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:42:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.56, 11.25, 10.92 [18:44:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.79, 11.89, 11.19 [18:45:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.30, 11.37, 11.28 [18:46:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.27, 11.17, 10.95 [18:46:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.47, 11.81, 11.25 [18:46:54] RECOVERY - test101 APT on test101 is OK: APT OK: 56 packages available for upgrade (0 critical updates). [18:47:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.55, 9.75, 9.72 [18:49:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.06, 11.64, 11.45 [18:50:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.18, 11.83, 11.32 [18:50:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.49, 3.25 [18:51:26] PROBLEM - test101 APT on test101 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [18:52:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.67, 11.82, 11.35 [18:52:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.61, 10.91, 10.76 [18:52:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.78, 11.38, 11.22 [18:52:37] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.45, 3.09, 3.12 [18:54:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.15, 11.03, 10.83 [18:55:49] !log [universalomega@test101] finished deploy of {'l10n': True} to all - SUCCESS in 1555s [18:56:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.54, 11.80, 11.45 [18:56:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:56:08] !log [universalomega@mwtask111] finished deploy of {'l10n': True} to all - SUCCESS in 1557s [18:56:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:53] RECOVERY - test101 APT on test101 is OK: APT OK: 56 packages available for upgrade (0 critical updates). [18:58:01] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.30, 11.51, 11.06 [18:59:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.44, 11.26, 11.15 [19:00:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.04, 10.67, 10.82 [19:00:10] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/updateArticleCount.php --wiki=celebswiki --update (END - exit=0) [19:00:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:01:33] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.45, 3.42, 3.04 [19:01:47] [02mw-config] 07Universal-Omega closed pull request 03#4548: T9025: Add wgCosmosRailBlacklistedNamespaces to ManageWiki/namespaces - 13https://github.com/miraheze/mw-config/pull/4548 [19:01:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/93a761af6bfb...f152b0960037 [19:01:50] [02miraheze/mw-config] 07AgentIsai 03f152b09 - T9025: Add wgCosmosRailBlacklistedNamespaces to ManageWiki/namespaces (#4548) [19:02:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.28, 11.23, 11.24 [19:02:34] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [19:02:36] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 2s [19:02:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:02:56] miraheze/mw-config - Universal-Omega the build passed. [19:03:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:03:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.25, 2.98, 2.93 [19:04:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.80, 11.15, 11.19 [19:04:49] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f152b0960037...c033712bc447 [19:04:51] [02miraheze/mw-config] 07Universal-Omega 03c033712 - fix [19:04:54] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [19:04:56] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [19:05:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:05:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:05:58] miraheze/mw-config - Universal-Omega the build passed. [19:06:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.18, 10.27, 9.69 [19:06:25] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.22, 3.43, 3.06 [19:07:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.82, 11.96, 11.65 [19:08:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.71, 8.97, 9.28 [19:08:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.48, 3.12 [19:08:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.15, 11.33, 11.13 [19:08:41] PROBLEM - test101 APT on test101 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [19:09:54] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.54, 9.31, 10.10 [19:10:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.85, 11.45, 11.20 [19:10:50] RECOVERY - test101 APT on test101 is OK: APT OK: 56 packages available for upgrade (0 critical updates). [19:11:44] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [19:14:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.13, 3.79, 3.37 [19:14:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.36, 11.74, 11.38 [19:14:57] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.25, 10.75, 9.91 [19:16:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.70, 11.51, 11.35 [19:16:52] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.35, 10.07, 9.76 [19:17:30] PROBLEM - test101 APT on test101 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [19:18:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.19, 9.41, 10.14 [19:18:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.43, 10.54, 10.37 [19:18:52] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/commit/a91de37afe8a [19:18:53] [02miraheze/ManageWiki] 07Universal-Omega 03a91de37 - ManageWikiNamespaces: support NS_SPECIAL in overridedefault [19:18:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.63, 3.33, 3.12 [19:18:55] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ManageWiki [19:18:56] [02ManageWiki] 07Universal-Omega opened pull request 03#350: ManageWikiNamespaces: support NS_SPECIAL in overridedefault - 13https://github.com/miraheze/ManageWiki/pull/350 [19:19:05] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [19:19:07] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 2s [19:19:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:19:38] RECOVERY - test101 APT on test101 is OK: APT OK: 56 packages available for upgrade (0 critical updates). [19:19:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:20:09] !log [universalomega@test101] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [19:20:10] !log [universalomega@test101] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 1s [19:20:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:20:45] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.41, 11.10, 10.59 [19:20:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:20:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.77, 3.60, 3.23 [19:21:18] [02ManageWiki] 07Universal-Omega synchronize pull request 03#350: ManageWikiNamespaces: support NS_SPECIAL in overridedefault - 13https://github.com/miraheze/ManageWiki/pull/350 [19:21:19] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/a91de37afe8a...2ebf86749322 [19:21:21] [02miraheze/ManageWiki] 07Universal-Omega 032ebf867 - Update Hooks.php [19:21:26] !log [universalomega@test101] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [19:21:27] !log [universalomega@test101] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 0s [19:21:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:22:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:22:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.00, 10.99, 10.61 [19:22:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.75, 3.97, 3.63 [19:22:22] !log [@mwtask111] starting deploy of {'config': True} to all [19:22:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.16, 3.68, 3.19 [19:22:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:22:39] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 17s [19:22:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.48, 10.98, 10.62 [19:22:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:22:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.90, 3.61, 3.27 [19:23:30] [02ManageWiki] 07Universal-Omega closed pull request 03#350: ManageWikiNamespaces: support NS_SPECIAL in overridedefault - 13https://github.com/miraheze/ManageWiki/pull/350 [19:23:32] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/ebfcce9750ca...4fa5686bb271 [19:23:33] [02miraheze/ManageWiki] 07Universal-Omega 034fa5686 - ManageWikiNamespaces: support NS_SPECIAL in overridedefault (#350) [19:23:35] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ManageWiki [19:23:36] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:23:43] miraheze/ManageWiki - Universal-Omega the build passed. [19:24:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.59, 10.77, 10.58 [19:24:09] PROBLEM - test101 APT on test101 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [19:24:12] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.72, 4.20, 3.75 [19:24:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.94, 3.28, 3.10 [19:25:14] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/a1e72c203970...045cd7500436 [19:25:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.35, 11.75, 11.62 [19:25:15] !log [universalomega@test101] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [19:25:15] [02miraheze/mediawiki] 07Universal-Omega 03045cd75 - Update ManageWiki [19:25:18] !log [universalomega@test101] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 2s [19:25:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:35] !log [universalomega@mwtask111] starting deploy of {'world': True} to all [19:25:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:40] miraheze/ManageWiki - Universal-Omega the build passed. [19:25:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:49] !log [universalomega@test101] starting deploy of {'world': True} to all [19:25:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.60, 3.14, 3.17 [19:27:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.44, 10.71, 11.27 [19:28:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.51, 3.50, 3.58 [19:28:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.66, 10.72, 9.92 [19:28:32] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/4fa5686bb271...36b4c8ad2957 [19:28:33] [02miraheze/ManageWiki] 07Universal-Omega 0336b4c8a - add is_array check [19:28:46] miraheze/ManageWiki - Universal-Omega the build passed. [19:29:04] !log [universalomega@mwtask111] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [19:29:08] !log [universalomega@test101] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [19:29:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.81, 11.32, 11.40 [19:29:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:29:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:30:02] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/045cd7500436...85b16d5632e2 [19:30:03] [02miraheze/mediawiki] 07Universal-Omega 0385b16d5 - Update ManageWiki [19:30:07] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.67, 4.26, 3.82 [19:30:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.38, 10.71, 10.03 [19:30:21] !log [universalomega@test101] starting deploy of {'world': True} to all [19:30:26] !log [universalomega@mwtask111] starting deploy of {'world': True} to all [19:30:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:30:39] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.47, 9.79, 10.14 [19:30:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:32:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.46, 10.17, 9.93 [19:33:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.75, 11.65, 11.53 [19:33:28] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/36b4c8ad2957...08b45add0a15 [19:33:29] [02miraheze/ManageWiki] 07Universal-Omega 0308b45ad - move inside isset check [19:33:30] !log [universalomega@mwtask111] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [19:33:32] !log [universalomega@test101] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [19:33:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:03] miraheze/ManageWiki - Universal-Omega the build passed. [19:34:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.97, 3.82 [19:34:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:55] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/85b16d5632e2...a32eb8108e1f [19:34:57] [02miraheze/mediawiki] 07Universal-Omega 03a32eb81 - Update ManageWiki [19:34:58] RECOVERY - test101 APT on test101 is OK: APT OK: 56 packages available for upgrade (0 critical updates). [19:35:04] !log [universalomega@test101] starting deploy of {'world': True} to all [19:35:07] !log [universalomega@mwtask111] starting deploy of {'world': True} to all [19:35:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:35:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:36:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.66, 11.22, 10.71 [19:36:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.85, 4.59, 4.08 [19:36:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.21, 10.29, 10.07 [19:38:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.40, 10.38, 10.45 [19:38:35] miraheze/ManageWiki - Universal-Omega the build passed. [19:39:30] PROBLEM - test101 APT on test101 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [19:40:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.00, 10.04, 10.04 [19:40:30] !log [universalomega@test101] finished deploy of {'world': True} to all - SUCCESS in 326s [19:40:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:41:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.88, 11.38, 11.26 [19:41:29] !log [universalomega@mwtask111] finished deploy of {'world': True} to all - SUCCESS in 381s [19:41:39] RECOVERY - test101 APT on test101 is OK: APT OK: 56 packages available for upgrade (0 critical updates). [19:41:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:42:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.73, 9.64, 10.15 [19:42:24] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.20, 3.41, 3.25 [19:42:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.89, 11.75, 11.07 [19:42:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.20, 11.30, 10.21 [19:43:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.70, 3.84 [19:44:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.81, 10.57, 10.28 [19:44:18] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.37, 3.29, 3.22 [19:44:23] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.96, 11.33, 10.34 [19:44:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.06, 11.54, 11.08 [19:44:41] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/08b45add0a15...b5ed9142193b [19:44:43] [02miraheze/ManageWiki] 07Universal-Omega 03b5ed914 - Don't set if it already is set. [19:44:57] !log [universalomega@test101] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [19:45:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:45:19] !log [universalomega@test101] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 19s [19:45:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:45:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.66, 4.05, 3.95 [19:46:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.88, 10.17, 10.18 [19:46:11] PROBLEM - test101 APT on test101 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [19:47:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.03, 3.65, 3.82 [19:48:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.17, 3.77, 3.43 [19:48:30] !log [universalomega@mwtask111] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [19:48:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:47] !log [universalomega@mwtask111] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 17s [19:48:52] !log [universalomega@mwtask111] starting deploy of {'world': True} to all [19:48:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:59] !log [universalomega@test101] starting deploy of {'world': True} to all [19:49:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:13] !log [universalomega@test101] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [19:49:15] !log [universalomega@mwtask111] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [19:49:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:53] miraheze/ManageWiki - Universal-Omega the build passed. [19:50:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:11] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.37, 10.05, 10.09 [19:50:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.58, 11.49, 10.67 [19:50:37] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/a32eb8108e1f...9bfa56496a47 [19:50:38] [02miraheze/mediawiki] 07Universal-Omega 039bfa564 - Update ManageWiki [19:50:41] !log [universalomega@test101] starting deploy of {'world': True} to all [19:50:46] !log [universalomega@test101] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [19:50:49] !log [universalomega@test101] starting deploy of {'world': True} to all [19:50:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:53] !log [universalomega@mwtask111] starting deploy of {'world': True} to all [19:51:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.35, 11.80, 11.63 [19:51:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:51:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:51:45] !log [universalomega@test101] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [19:51:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.13, 10.79, 10.38 [19:52:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:16] !log [universalomega@test101] starting deploy of {'world': True} to all [19:52:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.43, 11.40, 10.74 [19:52:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:54:50] RECOVERY - test101 APT on test101 is OK: APT OK: 56 packages available for upgrade (0 critical updates). [19:55:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.11, 4.00, 3.77 [19:56:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.87, 8.95, 10.17 [19:57:39] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.47, 4.12, 3.84 [19:57:45] !log [universalomega@test101] finished deploy of {'world': True} to all - SUCCESS in 329s [19:57:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.66, 3.85, 3.81 [19:58:04] !log [universalomega@mwtask111] finished deploy of {'world': True} to all - SUCCESS in 431s [19:58:10] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:58:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:58:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:59:31] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:59:34] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.80, 3.82, 3.78 [19:59:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.55, 3.71 [20:00:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.37, 9.06, 9.85 [20:00:15] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.837 second response time [20:00:16] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.75, 9.62, 10.16 [20:01:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.75, 11.21, 11.07 [20:02:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.30, 9.96, 10.23 [20:03:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.45, 11.04, 11.04 [20:03:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/2fe69fd84e28 [20:03:30] [02miraheze/mw-config] 07Universal-Omega 032fe69fd - Use `$wgDiscordAdditionalIncomingWebhookUrls` in ManageWiki [20:03:31] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [20:03:33] [02mw-config] 07Universal-Omega opened pull request 03#4566: Use `$wgDiscordAdditionalIncomingWebhookUrls` in ManageWiki - 13https://github.com/miraheze/mw-config/pull/4566 [20:03:35] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.604 second response time [20:04:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2fe69fd84e28...a4a7bb914bb2 [20:04:22] [02miraheze/mw-config] 07Universal-Omega 03a4a7bb9 - Update Defines.php [20:04:23] [02mw-config] 07Universal-Omega synchronize pull request 03#4566: Use `$wgDiscordAdditionalIncomingWebhookUrls` in ManageWiki - 13https://github.com/miraheze/mw-config/pull/4566 [20:04:43] miraheze/mw-config - Universal-Omega the build passed. [20:05:17] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.96, 2.66, 3.27 [20:05:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a4a7bb914bb2...cee2daed610f [20:05:21] [02miraheze/mw-config] 07Universal-Omega 03cee2dae - Update LocalSettings.php [20:05:23] [02mw-config] 07Universal-Omega synchronize pull request 03#4566: Use `$wgDiscordAdditionalIncomingWebhookUrls` in ManageWiki - 13https://github.com/miraheze/mw-config/pull/4566 [20:05:33] miraheze/mw-config - Universal-Omega the build passed. [20:06:31] miraheze/mw-config - Universal-Omega the build passed. [20:07:10] [02mw-config] 07Universal-Omega edited pull request 03#4566: Use `$wgDiscordAdditionalIncomingWebhookUrls` in ManageWiki - 13https://github.com/miraheze/mw-config/pull/4566 [20:07:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.70, 3.09, 3.39 [20:10:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.34, 11.40, 10.79 [20:11:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.56, 3.91, 3.62 [20:12:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.56, 11.46, 10.89 [20:13:32] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/f2e000ad52a7 [20:13:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.94, 3.80, 3.62 [20:13:34] [02miraheze/mw-config] 07Universal-Omega 03f2e000a - Update LocalSettings.php [20:13:35] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [20:13:37] [02mw-config] 07Universal-Omega opened pull request 03#4567: Add `$wgCosmosEnabledRailModules['recentchanges']` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4567 [20:14:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.01, 11.75, 11.08 [20:14:46] miraheze/mw-config - Universal-Omega the build passed. [20:15:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.14, 11.45, 11.10 [20:16:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.86, 11.99, 11.26 [20:17:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f2e000ad52a7...52d385e0b569 [20:17:47] [02miraheze/mw-config] 07Universal-Omega 0352d385e - Update ManageWikiSettings.php [20:17:49] [02mw-config] 07Universal-Omega synchronize pull request 03#4567: Add `$wgCosmosEnabledRailModules['recentchanges']` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4567 [20:18:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.42, 12.06, 11.39 [20:18:55] miraheze/mw-config - Universal-Omega the build passed. [20:19:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.56, 11.52, 11.26 [20:19:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.86, 3.10, 3.34 [20:20:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.54, 10.33, 9.83 [20:21:44] !log [@test101] starting deploy of {'config': True} to all [20:21:45] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [20:21:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:22:00] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.54, 11.35, 10.26 [20:22:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.08, 11.77, 11.45 [20:22:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.59, 10.09, 9.80 [20:24:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.54, 11.94, 11.54 [20:25:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.45, 12.24, 11.50 [20:25:57] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.51, 11.31, 10.54 [20:26:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.80, 3.36, 3.13 [20:26:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.47, 11.05, 11.29 [20:27:04] [02mw-config] 07Universal-Omega synchronize pull request 03#4567: Add `$wgCosmosEnabledRailModules['recentchanges']` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4567 [20:27:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/52d385e0b569...57356597c88f [20:27:07] [02miraheze/mw-config] 07Universal-Omega 035735659 - Update ManageWikiSettings.php [20:28:06] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.82, 3.32, 3.16 [20:28:14] miraheze/mw-config - Universal-Omega the build passed. [20:31:53] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.34, 11.54, 10.84 [20:32:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/57356597c88f...556c03477ba4 [20:32:15] [02miraheze/mw-config] 07Universal-Omega 03556c034 - Update ManageWikiSettings.php [20:32:17] [02mw-config] 07Universal-Omega synchronize pull request 03#4567: Add `$wgCosmosEnabledRailModules['recentchanges']` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4567 [20:33:18] miraheze/mw-config - Universal-Omega the build passed. [20:33:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.35, 9.99, 9.77 [20:33:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.36, 11.44, 10.89 [20:34:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:35:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.20, 11.37, 11.55 [20:36:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.75, 3.48 [20:36:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.44, 10.07, 9.48 [20:38:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.28, 10.32, 9.79 [20:38:25] [02ManageWiki] 07Universal-Omega synchronize pull request 03#349: Support associative arrays in ManageWikiSettings - 13https://github.com/miraheze/ManageWiki/pull/349 [20:38:26] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/c1ab292123db...045ff755fcee [20:38:28] [02miraheze/ManageWiki] 07Universal-Omega 03045ff75 - Update ManageWikiFormFactoryBuilder.php [20:39:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:40:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.73, 4.01, 3.62 [20:40:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.13, 11.23, 10.98 [20:40:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.36, 10.08, 9.63 [20:41:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.50, 11.20, 10.84 [20:42:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.62, 3.69, 3.53 [20:42:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.32, 11.28, 11.04 [20:43:11] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.58, 10.13, 10.06 [20:43:49] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.19, 10.91, 10.77 [20:43:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.51, 9.93, 9.85 [20:44:11] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.11, 4.02, 3.66 [20:44:21] miraheze/ManageWiki - Universal-Omega the build passed. [20:44:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.04, 10.69, 9.97 [20:46:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.70, 3.59 [20:46:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.43, 3.18 [20:48:08] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.65, 4.14, 3.74 [20:48:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.64, 9.96, 9.84 [20:48:44] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.91, 3.39 [20:49:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.28, 11.13, 10.84 [20:50:28] !log [@test101] starting deploy of {'config': True} to all [20:50:29] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [20:50:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:50:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.64, 3.35 [20:50:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:51:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.53, 10.79, 10.77 [20:52:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.82, 3.89, 3.71 [20:52:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.89, 3.37, 3.29 [20:52:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.99, 11.23, 10.45 [20:54:01] [02ManageWiki] 07Universal-Omega synchronize pull request 03#349: Support associative arrays in ManageWikiSettings - 13https://github.com/miraheze/ManageWiki/pull/349 [20:54:02] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/045ff755fcee...4183a1c1ef85 [20:54:04] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.94, 3.75 [20:54:04] [02miraheze/ManageWiki] 07Universal-Omega 034183a1c - Update ManageWikiFormFactoryBuilder.php [20:54:22] [02mw-config] 07Universal-Omega synchronize pull request 03#4567: Add `$wgCosmosEnabledRailModules['recentchanges']` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4567 [20:54:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/556c03477ba4...2736765decbc [20:54:25] [02miraheze/mw-config] 07Universal-Omega 032736765 - Update ManageWikiSettings.php [20:55:25] miraheze/mw-config - Universal-Omega the build passed. [20:55:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.70, 3.98, 3.44 [20:56:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.76, 3.83, 3.51 [20:57:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 4.00, 3.88, 3.46 [20:59:00] miraheze/ManageWiki - Universal-Omega the build passed. [20:59:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.43, 3.32, 3.30 [20:59:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.15, 3.72, 3.82 [21:01:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.60, 11.12, 11.01 [21:02:13] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/4183a1c1ef85...82fb0571d1f3 [21:02:15] [02miraheze/ManageWiki] 07Universal-Omega 0382fb057 - Update ManageWikiFormFactoryBuilder.php [21:02:16] [02ManageWiki] 07Universal-Omega synchronize pull request 03#349: Support associative arrays in ManageWikiSettings - 13https://github.com/miraheze/ManageWiki/pull/349 [21:02:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.08, 2.95, 3.22 [21:03:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.52, 1.65, 2.00 [21:03:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.97, 10.99, 10.97 [21:07:58] miraheze/ManageWiki - Universal-Omega the build passed. [21:09:52] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.05, 3.10, 3.40 [21:11:03] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.13, 1.27, 1.67 [21:11:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.86, 11.07, 10.70 [21:12:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.95, 9.97, 9.43 [21:12:26] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.84, 10.30, 9.89 [21:13:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.07, 10.62, 10.61 [21:14:00] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.69, 9.20, 9.23 [21:14:55] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.82, 1.93, 1.19 [21:16:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.03, 10.74, 10.84 [21:16:55] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.02, 1.61, 1.16 [21:17:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.65, 9.58, 10.19 [21:18:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.32, 9.63, 9.79 [21:18:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.25, 10.00, 10.58 [21:19:15] [02ManageWiki] 07Universal-Omega synchronize pull request 03#349: Support associative arrays in ManageWikiSettings - 13https://github.com/miraheze/ManageWiki/pull/349 [21:19:16] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/82fb0571d1f3...909868fa50e7 [21:19:18] [02miraheze/ManageWiki] 07Universal-Omega 03909868f - Update ManageWikiFormFactoryBuilder.php [21:21:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.61, 9.52, 10.16 [21:21:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.31, 10.55, 10.45 [21:23:51] miraheze/ManageWiki - Universal-Omega the build passed. [21:26:02] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.19, 9.67, 10.15 [21:26:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.52, 9.16, 10.11 [21:26:47] [02ManageWiki] 07Universal-Omega edited pull request 03#349: Support associative arrays in ManageWikiSettings - 13https://github.com/miraheze/ManageWiki/pull/349 [21:26:57] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.56, 11.52, 10.49 [21:28:43] !log [universalomega@test101] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [21:28:44] !log [universalomega@test101] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 1s [21:28:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.13, 11.22, 10.50 [21:28:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:29:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:30:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.21, 10.56, 10.44 [21:30:08] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.97, 11.49, 10.69 [21:30:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.69, 10.58, 10.49 [21:33:59] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.18, 11.86, 11.07 [21:34:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.95, 9.63, 10.08 [21:34:36] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.29, 12.08, 10.98 [21:35:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 10.00, 9.70, 10.12 [21:35:55] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.87, 12.10, 11.24 [21:36:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.02, 9.75, 10.13 [21:36:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.11, 10.90, 10.68 [21:39:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.80, 11.49, 11.20 [21:42:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.33, 9.40, 10.03 [21:44:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.43, 10.69, 10.41 [21:46:20] [02ManageWiki] 07Universal-Omega synchronize pull request 03#349: Support associative arrays in ManageWikiSettings - 13https://github.com/miraheze/ManageWiki/pull/349 [21:46:22] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/909868fa50e7...d033c7111d88 [21:46:23] [02miraheze/ManageWiki] 07Universal-Omega 03d033c71 - Update ManageWikiFormFactoryBuilder.php [21:47:01] [02ManageWiki] 07Universal-Omega synchronize pull request 03#349: Support associative arrays in ManageWikiSettings - 13https://github.com/miraheze/ManageWiki/pull/349 [21:47:03] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/d033c7111d88...d2e287849bfc [21:47:05] [02miraheze/ManageWiki] 07Universal-Omega 03d2e2878 - Update ManageWikiFormFactoryBuilder.php [21:48:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.48, 9.69, 10.10 [21:49:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:50:33] [02mw-config] 07Universal-Omega synchronize pull request 03#4567: Add `$wgCosmosEnabledRailModules['recentchanges']` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4567 [21:50:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2736765decbc...8ec7f20c15bf [21:50:36] [02miraheze/mw-config] 07Universal-Omega 038ec7f20 - Update ManageWikiSettings.php [21:51:29] miraheze/ManageWiki - Universal-Omega the build passed. [21:51:44] miraheze/mw-config - Universal-Omega the build passed. [21:52:55] [02ManageWiki] 07Universal-Omega closed pull request 03#349: Support associative arrays in ManageWikiSettings - 13https://github.com/miraheze/ManageWiki/pull/349 [21:52:57] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/b5ed9142193b...dc14eb5080bf [21:52:58] [02miraheze/ManageWiki] 07Universal-Omega 03dc14eb5 - Support associative arrays in ManageWikiSettings (#349) [21:53:00] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/ManageWiki [21:53:01] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [21:53:18] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.05, 10.87, 10.70 [21:53:52] miraheze/ManageWiki - Universal-Omega the build passed. [21:54:22] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/9bfa56496a47...3fc2655794ac [21:54:24] [02miraheze/mediawiki] 07Universal-Omega 033fc2655 - Update ManageWiki [21:56:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:56:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.95, 10.37, 10.23 [21:56:27] [02miraheze/ManageWiki] 07Universal-Omega pushed 038 commits to 03Universal-Omega-patch-2 [+0/-0/±16] 13https://github.com/miraheze/ManageWiki/compare/4ccb4f92818e...4cb3ea8fc475 [21:56:29] [02miraheze/ManageWiki] 07Universal-Omega 034cb3ea8 - Merge branch 'master' into Universal-Omega-patch-2 [21:56:30] [02ManageWiki] 07Universal-Omega synchronize pull request 03#348: Fix users and wikipages type to always be array, not string - 13https://github.com/miraheze/ManageWiki/pull/348 [21:57:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.92, 10.72, 10.70 [22:00:44] miraheze/ManageWiki - Universal-Omega the build passed. [22:01:09] miraheze/ManageWiki - Universal-Omega the build passed. [22:01:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.72, 11.90, 11.12 [22:01:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.68, 10.16, 9.72 [22:02:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.12, 11.55, 10.83 [22:04:01] [02ManageWiki] 07Universal-Omega synchronize pull request 03#348: Fix users and wikipages type to always be array, not string - 13https://github.com/miraheze/ManageWiki/pull/348 [22:04:02] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/4cb3ea8fc475...ca9e3361639d [22:04:04] [02miraheze/ManageWiki] 07Universal-Omega 03ca9e336 - Update ManageWikiTypes.php [22:04:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.83, 11.10, 10.77 [22:04:48] !log [universalomega@test101] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [22:04:49] !log [universalomega@test101] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 0s [22:04:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:05:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:05:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.48, 11.27, 11.10 [22:07:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.06, 8.84, 9.42 [22:09:13] miraheze/ManageWiki - Universal-Omega the build passed. [22:10:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.40, 10.71, 10.60 [22:11:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.01, 10.29, 9.41 [22:12:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.03, 10.91, 10.70 [22:13:11] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/ca9e3361639d...0240aececb6b [22:13:12] [02miraheze/ManageWiki] 07Universal-Omega 030240aec - Update ManageWikiTypes.php [22:13:14] [02ManageWiki] 07Universal-Omega synchronize pull request 03#348: Fix users and wikipages type to always be array, not string - 13https://github.com/miraheze/ManageWiki/pull/348 [22:13:20] !log [universalomega@test101] starting deploy of {'folders': 'w/extensions/ManageWiki'} to all [22:13:21] !log [universalomega@test101] finished deploy of {'folders': 'w/extensions/ManageWiki'} to all - SUCCESS in 0s [22:13:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:13:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.34, 10.35, 9.53 [22:13:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:15:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.80, 10.03, 9.51 [22:15:39] [02ManageWiki] 07Universal-Omega closed pull request 03#348: Fix users and wikipages type to always be array, not string - 13https://github.com/miraheze/ManageWiki/pull/348 [22:15:40] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/compare/dc14eb5080bf...59c9a041359c [22:15:42] [02miraheze/ManageWiki] 07Universal-Omega 0359c9a04 - Fix users and wikipages type to always be array, not string (#348) [22:15:43] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/ManageWiki [22:15:45] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [22:17:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.98, 1.60, 1.36 [22:17:27] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/3fc2655794ac...8b96974a1d3c [22:17:29] [02miraheze/mediawiki] 07Universal-Omega 038b96974 - Update ManageWiki [22:17:47] !log [universalomega@test101] starting deploy of {'world': True} to all [22:17:52] miraheze/ManageWiki - Universal-Omega the build passed. [22:17:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:18:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.74, 3.88, 3.39 [22:19:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.45, 2.14, 1.58 [22:19:46] !log [universalomega@mwtask111] starting deploy of {'world': True} to all [22:19:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:20:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.83, 11.38, 10.90 [22:20:53] miraheze/ManageWiki - Universal-Omega the build passed. [22:22:18] !log [universalomega@test101] finished deploy of {'world': True} to all - SUCCESS in 271s [22:22:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:22:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.84, 10.92, 10.77 [22:22:51] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.26, 3.39, 3.33 [22:23:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.27, 11.67, 11.01 [22:25:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.19, 1.83, 1.67 [22:25:11] !log [universalomega@test101] starting deploy of {'config': True} to all [22:25:12] !log [universalomega@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [22:25:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.85, 11.19, 10.91 [22:25:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:25:29] !log [universalomega@mwtask111] finished deploy of {'world': True} to all - SUCCESS in 343s [22:25:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:25:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:27:03] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.28, 1.65, 1.62 [22:30:23] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.77, 1.50, 1.15 [22:32:23] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.39, 1.38, 1.14 [22:32:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.36, 9.41, 10.17 [22:40:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.33, 10.26, 10.19 [22:41:59] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.90, 1.65, 1.54 [22:42:27] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.68, 9.54, 9.95 [22:43:08] [02mw-config] 07Universal-Omega edited pull request 03#4566: Use `$wgDiscordAdditionalIncomingWebhookUrls` in ManageWiki - 13https://github.com/miraheze/mw-config/pull/4566 [22:43:09] [02mw-config] 07Universal-Omega edited pull request 03#4566: Use `$wgDiscordAdditionalIncomingWebhookUrls` in ManageWiki - 13https://github.com/miraheze/mw-config/pull/4566 [22:43:48] [02mw-config] 07Universal-Omega edited pull request 03#4566: Use `$wgDiscordAdditionalIncomingWebhookUrls` in ManageWiki - 13https://github.com/miraheze/mw-config/pull/4566 [22:43:49] [02mw-config] 07Universal-Omega edited pull request 03#4566: Use `$wgDiscordAdditionalIncomingWebhookUrls` in ManageWiki - 13https://github.com/miraheze/mw-config/pull/4566 [22:43:55] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.42, 1.51, 1.51 [22:44:24] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.37, 3.14 [22:45:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.51, 3.93, 3.38 [22:47:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.71, 3.37 [22:48:13] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.53, 3.19, 3.15 [22:48:59] !log [@test101] starting deploy of {'config': True} to all [22:49:00] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [22:49:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:49:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.08, 10.81, 10.51 [22:49:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:49:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.63, 3.32, 3.27 [22:49:48] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.72, 1.60 [22:49:50] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.16, 10.51, 9.74 [22:51:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.54, 10.33, 10.38 [22:51:21] [02mw-config] 07Universal-Omega commented on pull request 03#4566: Use `$wgDiscordAdditionalIncomingWebhookUrls` in ManageWiki - 13https://github.com/miraheze/mw-config/pull/4566#issuecomment-1097310297 [22:51:22] [02mw-config] 07Universal-Omega commented on pull request 03#4566: Use `$wgDiscordAdditionalIncomingWebhookUrls` in ManageWiki - 13https://github.com/miraheze/mw-config/pull/4566#issuecomment-1097310295 [22:51:39] [02mw-config] 07Universal-Omega deleted a comment on pull request 03#4566: Use `$wgDiscordAdditionalIncomingWebhookUrls` in ManageWiki - 13https://github.com/miraheze/mw-config/pull/4566#issuecomment-1097310297 [22:51:44] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.16, 1.53, 1.54 [22:51:49] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.87, 9.64, 9.51 [22:53:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.83, 3.54 [22:53:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.65, 3.43, 3.15 [22:55:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.54, 2.78, 2.94 [22:57:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.89, 9.77, 10.14 [22:57:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.45, 3.15, 3.32 [23:00:02] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:03] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:03:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.52, 11.32, 10.67 [23:04:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.96, 10.32, 9.33 [23:07:41] [02mw-config] 07Universal-Omega closed pull request 03#4566: Use `$wgDiscordAdditionalIncomingWebhookUrls` in ManageWiki - 13https://github.com/miraheze/mw-config/pull/4566 [23:07:42] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [23:07:44] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [23:07:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/mw-config/compare/c033712bc447...9b84500365f3 [23:07:47] [02miraheze/mw-config] 07Universal-Omega 039b84500 - Use `$wgDiscordAdditionalIncomingWebhookUrls` in ManageWiki (#4566) [23:07:52] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [23:08:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:08:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:08:20] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s [23:08:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:08:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.42, 9.32, 9.18 [23:08:42] miraheze/mw-config - Universal-Omega the build passed. [23:09:11] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.09, 11.35, 10.28 [23:09:59] [02mw-config] 07Universal-Omega edited a comment on pull request 03#4566: Use `$wgDiscordAdditionalIncomingWebhookUrls` in ManageWiki - 13https://github.com/miraheze/mw-config/pull/4566#issuecomment-1097310295 [23:10:06] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.36, 2.45, 1.61 [23:11:08] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.89, 11.20, 10.33 [23:12:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.09, 1.87, 1.49 [23:13:04] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.31, 11.46, 10.53 [23:13:12] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.07, 1.71, 1.56 [23:13:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.14, 9.40, 10.00 [23:13:32] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 810s [23:13:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:14:04] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.15, 1.58, 1.43 [23:14:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.65, 10.60, 9.69 [23:14:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.26, 9.90, 9.26 [23:14:55] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.23, 1.71, 1.15 [23:15:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.68, 11.29, 10.59 [23:16:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.44, 10.72, 9.84 [23:16:55] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.68, 1.28, 1.06 [23:17:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.00, 9.98, 10.07 [23:17:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 3.60, 3.15 [23:18:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.52, 11.34, 10.17 [23:18:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.46, 3.31 [23:18:54] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.99, 10.07, 10.19 [23:19:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.80, 9.75, 9.99 [23:19:46] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.04, 3.36, 3.11 [23:20:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.08, 10.70, 10.09 [23:20:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.52, 3.84, 3.45 [23:21:36] [02mw-config] 07Universal-Omega closed pull request 03#4565: test: Validate ManageWikiSettings.php file - 13https://github.com/miraheze/mw-config/pull/4565 [23:21:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+2/-0/±2] 13https://github.com/miraheze/mw-config/compare/9b84500365f3...b0503f6279ae [23:21:39] [02miraheze/mw-config] 07lens0021 03b0503f6 - test: Validate ManageWikiSettings.php file (#4565) [23:21:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.78, 10.94, 9.84 [23:21:57] !log [@test101] starting deploy of {'config': True} to all [23:21:59] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [23:22:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.91, 3.45, 3.36 [23:22:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:45] miraheze/mw-config - Universal-Omega the build passed. [23:23:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:23:15] !log [@mwtask111] starting deploy of {'config': True} to all [23:23:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:23:25] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 10s [23:23:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:24:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.77, 9.75, 9.85 [23:24:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.64, 3.18, 3.26 [23:25:34] [02miraheze/mw-config] 07Universal-Omega pushed 033 commits to 03Universal-Omega-patch-2 [+4/-0/±9] 13https://github.com/miraheze/mw-config/compare/8ec7f20c15bf...bf58c98dddaf [23:25:35] [02miraheze/mw-config] 07Universal-Omega 03bf58c98 - Merge branch 'master' into Universal-Omega-patch-2 [23:25:37] [02mw-config] 07Universal-Omega synchronize pull request 03#4567: Add `$wgCosmosEnabledRailModules['recentchanges']` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4567 [23:26:17] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.11, 9.66, 9.60 [23:26:18] !log [@test101] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1576s [23:26:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:26:37] miraheze/mw-config - Universal-Omega the build has errored. [23:27:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.51, 9.63, 9.61 [23:28:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/bf58c98dddaf...869c5fbe00bb [23:28:54] [02miraheze/mw-config] 07Universal-Omega 03869c5fb - Update SettingsTest.php [23:28:55] [02mw-config] 07Universal-Omega synchronize pull request 03#4567: Add `$wgCosmosEnabledRailModules['recentchanges']` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4567 [23:29:58] miraheze/mw-config - Universal-Omega the build passed. [23:30:10] [02mw-config] 07Universal-Omega closed pull request 03#4567: Add `$wgCosmosEnabledRailModules['recentchanges']` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4567 [23:30:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/mw-config/compare/b0503f6279ae...e1d78c042bba [23:30:13] [02miraheze/mw-config] 07Universal-Omega 03e1d78c0 - Add `$wgCosmosEnabledRailModules['recentchanges']` to ManageWikiSettings (#4567) [23:30:15] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [23:30:16] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [23:31:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.23, 1.75, 1.94 [23:31:16] miraheze/mw-config - Universal-Omega the build passed. [23:35:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:37:03] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.25, 1.37, 1.69 [23:38:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.39, 10.79, 9.97 [23:40:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.82, 10.82, 10.08 [23:41:14] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [23:41:14] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:41:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.56, 3.70, 3.39 [23:42:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.13, 11.26, 10.41 [23:43:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/8ea148f4fae7 [23:43:02] [02miraheze/mw-config] 07Universal-Omega 038ea148f - undo hack for wikipages and users type configs [23:43:03] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [23:43:10] [02mw-config] 07Universal-Omega opened pull request 03#4568: undo hack for wikipages and users type configs - 13https://github.com/miraheze/mw-config/pull/4568 [23:43:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.59, 3.95, 3.51 [23:44:14] miraheze/mw-config - Universal-Omega the build passed. [23:44:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.80, 11.08, 10.43 [23:45:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.56, 3.41 [23:47:50] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.63, 3.79, 3.49 [23:48:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.11, 9.81, 10.07 [23:49:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8ea148f4fae7...9cb394765bd6 [23:49:24] [02miraheze/mw-config] 07Universal-Omega 039cb3947 - Update ManageWikiSettings.php [23:49:26] [02mw-config] 07Universal-Omega synchronize pull request 03#4568: undo hack for wikipages and users type configs - 13https://github.com/miraheze/mw-config/pull/4568 [23:49:35] !log [@test101] starting deploy of {'config': True} to all [23:49:36] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [23:49:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.51, 10.07, 9.15 [23:49:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:49:48] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.08, 3.38, 3.37 [23:49:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:50:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:50:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9cb394765bd6...f2ee436bc558 [23:50:19] [02miraheze/mw-config] 07Universal-Omega 03f2ee436 - Update LocalSettings.php [23:50:21] [02mw-config] 07Universal-Omega synchronize pull request 03#4568: undo hack for wikipages and users type configs - 13https://github.com/miraheze/mw-config/pull/4568 [23:50:27] miraheze/mw-config - Universal-Omega the build passed. [23:51:22] miraheze/mw-config - Universal-Omega the build passed. [23:51:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.35, 10.41, 10.01 [23:52:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.69, 10.50, 10.31 [23:52:28] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.65, 11.05, 10.56 [23:53:07] !log [@mwtask111] starting deploy of {'config': True} to all [23:53:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:53:25] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 18s [23:53:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.34, 11.25, 10.35 [23:53:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:55:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.22, 10.98, 9.82 [23:57:22] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.23, 11.34, 10.62 [23:57:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:57:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.14, 11.03, 9.98 [23:58:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.97, 10.66, 10.67 [23:58:51] [02mw-config] 07Universal-Omega edited pull request 03#4568: undo hack for wikipages and users type configs - 13https://github.com/miraheze/mw-config/pull/4568 [23:59:18] [02mw-config] 07Universal-Omega edited pull request 03#4568: undo hack for wikipages and users type configs - 13https://github.com/miraheze/mw-config/pull/4568 [23:59:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.75, 3.76, 3.50