[00:00:20] [02mediawiki] 07github-actions[bot] labeled pull request 03#6143: Bump extensions/FlaggedRevs from `9db1af1` to `f8d3301` - 13https://github.com/miraheze/mediawiki/pull/6143 [00:01:27] [02mediawiki] 07dependabot[bot] commented on pull request 03#5716: Bump skins/MonoBook from `04ddc68` to `82cf50f` - 13https://github.com/miraheze/mediawiki/pull/5716#issuecomment-1085247018 [00:01:41] [02mediawiki] 07dependabot[bot] commented on pull request 03#5829: Bump extensions/ArticlePlaceholder from `f17427e` to `8d6e792` - 13https://github.com/miraheze/mediawiki/pull/5829#issuecomment-1085247170 [00:01:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.02, 9.02, 9.43 [00:03:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.72, 3.65, 3.39 [00:03:19] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 707 failures. Last run 2 minutes ago with 707 failures. Failed resources (up to 3 shown): File[kunwok.org],File[kunwok.org_private],File[www.lab612.at],File[www.lab612.at_private] [00:03:19] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.96, 1.61, 1.28 [00:03:22] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.94, 1.85, 1.63 [00:03:33] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 3.63, 3.37 [00:04:04] [02mediawiki] 07dependabot[bot] commented on pull request 03#5762: Bump extensions/CheckUser from `c147485` to `7b1eb4b` - 13https://github.com/miraheze/mediawiki/pull/5762#issuecomment-1085248999 [00:04:13] PROBLEM - mem121 Puppet on mem121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:05:13] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.66, 1.62, 1.32 [00:05:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.65, 4.09, 3.51 [00:05:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.23, 9.83, 9.31 [00:06:01] [02mediawiki] 07dependabot[bot] commented on pull request 03#5861: Bump extensions/PageForms from `1d74db6` to `324cceb` - 13https://github.com/miraheze/mediawiki/pull/5861#issuecomment-1085250553 [00:06:52] [02mediawiki] 07dependabot[bot] commented on pull request 03#5855: Bump extensions/SpriteSheet from `e8f498a` to `e3d84c0` - 13https://github.com/miraheze/mediawiki/pull/5855#issuecomment-1085251300 [00:06:58] [02mediawiki] 07dependabot[bot] commented on pull request 03#5846: Bump extensions/ManageWiki from `35b5466` to `b57a1a1` - 13https://github.com/miraheze/mediawiki/pull/5846#issuecomment-1085251385 [00:07:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.61, 1.70, 1.61 [00:07:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.36, 9.68, 9.32 [00:07:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.62, 8.93, 9.04 [00:08:42] [02mediawiki] 07dependabot[bot] commented on pull request 03#5673: Bump extensions/GlobalWatchlist from `4cff4b3` to `8f3f6f5` - 13https://github.com/miraheze/mediawiki/pull/5673#issuecomment-1085252769 [00:08:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.34, 9.07, 9.15 [00:09:42] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.04, 8.96, 9.05 [00:09:44] [02mediawiki] 07dependabot[bot] commented on pull request 03#5850: Bump extensions/Babel from `265a94d` to `adeb39a` - 13https://github.com/miraheze/mediawiki/pull/5850#issuecomment-1085253477 [00:10:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.27, 9.09, 9.15 [00:11:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.29, 10.98, 9.87 [00:11:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.65, 9.81, 9.36 [00:13:25] [02mediawiki] 07dependabot[bot] commented on pull request 03#5670: Bump extensions/CentralAuth from `306fa3a` to `6fe770b` - 13https://github.com/miraheze/mediawiki/pull/5670#issuecomment-1085256058 [00:13:38] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.17, 9.01, 9.12 [00:15:25] [02mediawiki] 07dependabot[bot] commented on pull request 03#5658: Bump extensions/PageTriage from `914c4b1` to `bd9cbb8` - 13https://github.com/miraheze/mediawiki/pull/5658#issuecomment-1085257175 [00:15:35] [02mediawiki] 07dependabot[bot] commented on pull request 03#5681: Bump extensions/OAuth from `6baeee2` to `6048fc3` - 13https://github.com/miraheze/mediawiki/pull/5681#issuecomment-1085257269 [00:15:49] [02mediawiki] 07dependabot[bot] commented on pull request 03#5861: Bump extensions/PageForms from `1d74db6` to `324cceb` - 13https://github.com/miraheze/mediawiki/pull/5861#issuecomment-1085257416 [00:16:05] [02mediawiki] 07dependabot[bot] commented on pull request 03#5846: Bump extensions/ManageWiki from `35b5466` to `b57a1a1` - 13https://github.com/miraheze/mediawiki/pull/5846#issuecomment-1085257775 [00:16:28] [02mediawiki] 07dependabot[bot] commented on pull request 03#5663: Bump extensions/TranslationNotifications from `7956b32` to `373b735` - 13https://github.com/miraheze/mediawiki/pull/5663#issuecomment-1085258361 [00:17:34] [02mediawiki] 07dependabot[bot] commented on pull request 03#5660: Bump extensions/SyntaxHighlight_GeSHi from `c26f9a2` to `318bbe5` - 13https://github.com/miraheze/mediawiki/pull/5660#issuecomment-1085259135 [00:17:36] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.24, 1.88, 1.58 [00:19:30] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.59, 1.77, 1.57 [00:19:31] !log [macfan@test101] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 1229s [00:19:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.84, 9.45, 9.12 [00:19:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:19:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.94, 7.51, 8.38 [00:20:07] [02mediawiki] 07dependabot[bot] commented on pull request 03#5671: Bump extensions/ManageWiki from `35b5466` to `c1e596e` - 13https://github.com/miraheze/mediawiki/pull/5671#issuecomment-1085260518 [00:20:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.53, 7.47, 8.36 [00:21:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.96, 3.90 [00:23:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.39, 3.72, 3.89 [00:23:19] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.35, 1.60, 1.56 [00:24:10] [02mediawiki] 07dependabot[bot] commented on pull request 03#5661: Bump extensions/UniversalLanguageSelector from `32ee5f5` to `f8eda0c` - 13https://github.com/miraheze/mediawiki/pull/5661#issuecomment-1085262534 [00:24:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.31, 3.36, 3.91 [00:26:06] !log [macfan@test101] starting deploy of {'world': True} to all [00:26:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.69, 3.80, 3.85 [00:27:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.00, 9.07, 9.27 [00:27:45] [02mediawiki] 07dependabot[bot] commented on pull request 03#5817: Bump skins/apex from `5448eaf` to `4b8e3ca` - 13https://github.com/miraheze/mediawiki/pull/5817#issuecomment-1085264046 [00:27:59] [02mediawiki] 07dependabot[bot] commented on pull request 03#5833: Bump extensions/Linter from `5b70116` to `96a903f` - 13https://github.com/miraheze/mediawiki/pull/5833#issuecomment-1085264138 [00:28:39] !log aborted deploy to mw* while i work on fixing beta [00:28:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:28:57] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.29, 1.73, 1.48 [00:29:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.56, 3.78, 3.85 [00:30:57] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.43, 1.25, 1.33 [00:31:26] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [00:31:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.60, 8.77, 9.68 [00:32:13] RECOVERY - mem121 Puppet on mem121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:32:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 3.88, 3.84 [00:33:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.89, 4.00, 3.90 [00:33:24] [02mediawiki] 07dependabot[bot] commented on pull request 03#5775: Bump extensions/ProofreadPage from `866ad8a` to `720790c` - 13https://github.com/miraheze/mediawiki/pull/5775#issuecomment-1085266488 [00:33:27] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 3.66, 3.67 [00:33:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.60, 9.17, 9.69 [00:34:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.72, 3.79 [00:34:58] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.87, 3.86, 3.85 [00:35:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.84, 8.73, 8.43 [00:37:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.44, 9.11, 9.65 [00:38:48] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.27, 9.39, 8.53 [00:39:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.72, 3.89, 3.79 [00:41:05] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.26, 9.42, 8.82 [00:41:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.67, 9.42, 9.20 [00:42:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.19, 9.86, 8.96 [00:43:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.25, 8.99, 8.73 [00:43:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.97, 9.79, 9.37 [00:43:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.09, 4.18, 3.87 [00:44:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.94, 3.08, 3.40 [00:45:29] [02mediawiki] 07dependabot[bot] commented on pull request 03#5710: Bump extensions/UrlShortener from `e93b161` to `fa0b0de` - 13https://github.com/miraheze/mediawiki/pull/5710#issuecomment-1085272523 [00:46:16] PROBLEM - test101 Disk Space on test101 is WARNING: DISK WARNING - free space: / 1347 MB (7% inode=55%); [00:46:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.37, 9.98, 9.18 [00:46:52] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.36, 9.58, 9.00 [00:46:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 3.39, 3.52 [00:48:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.73, 9.83, 9.21 [00:48:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 3.99, 3.72 [00:48:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.77, 9.40, 9.02 [00:48:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.00, 3.41, 3.52 [00:49:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.14, 10.68, 9.82 [00:49:23] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.09, 3.93, 3.92 [00:50:13] [02mediawiki] 07dependabot[bot] commented on pull request 03#5850: Bump extensions/Babel from `265a94d` to `adeb39a` - 13https://github.com/miraheze/mediawiki/pull/5850#issuecomment-1085274642 [00:50:14] [02mediawiki] 07dependabot[bot] commented on pull request 03#5855: Bump extensions/SpriteSheet from `e8f498a` to `e3d84c0` - 13https://github.com/miraheze/mediawiki/pull/5855#issuecomment-1085274640 [00:50:42] [02mediawiki] 07dependabot[bot] commented on pull request 03#5856: Bump vendor from `56c2e8a` to `ad86647` - 13https://github.com/miraheze/mediawiki/pull/5856#issuecomment-1085274880 [00:50:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.80, 3.67 [00:50:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.92, 2.87, 3.31 [00:51:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.00, 9.13, 9.09 [00:52:41] [02mediawiki] 07dependabot[bot] commented on pull request 03#5829: Bump extensions/ArticlePlaceholder from `f17427e` to `8d6e792` - 13https://github.com/miraheze/mediawiki/pull/5829#issuecomment-1085275828 [00:52:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.17, 9.31, 9.02 [00:53:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.95, 9.49, 9.23 [00:54:12] !log [macfan@test101] starting deploy of {'world': True} to all [00:54:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:54:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.96, 9.08, 8.97 [00:54:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.70, 2.91, 3.36 [00:56:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.72, 9.45, 9.10 [00:56:50] [02mediawiki] 07dependabot[bot] commented on pull request 03#5705: Bump extensions/MassEditRegex from `514a345` to `d5c58d7` - 13https://github.com/miraheze/mediawiki/pull/5705#issuecomment-1085280616 [00:57:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.04, 4.49, 4.08 [00:57:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.24, 9.74, 9.37 [00:58:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.48, 8.82, 8.90 [00:59:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.20, 9.71, 9.82 [00:59:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.66, 9.62, 9.37 [00:59:43] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.41, 3.35 [00:59:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.05, 8.90, 8.87 [01:00:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.56, 3.81, 3.60 [01:01:07] !log [macfan@test101] finished deploy of {'world': True} to all - SUCCESS in 415s [01:01:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.06, 3.90, 3.96 [01:01:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:01:41] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.06, 3.32, 3.33 [01:01:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.45, 8.72, 8.78 [01:02:21] [02mediawiki] 07dependabot[bot] commented on pull request 03#5731: Bump extensions/ImageRating from `4486d35` to `83b046b` - 13https://github.com/miraheze/mediawiki/pull/5731#issuecomment-1085284656 [01:02:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.69, 3.27, 3.42 [01:03:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.10, 9.38, 9.30 [01:04:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.66, 9.60, 9.19 [01:04:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.70, 3.21, 3.39 [01:05:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.16, 4.27, 4.05 [01:05:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:06:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.77, 9.71, 9.29 [01:07:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.81, 9.90, 9.53 [01:07:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.72, 8.04, 8.44 [01:08:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.11, 3.69, 3.54 [01:10:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.67, 10.02, 9.49 [01:10:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.63, 3.76, 3.59 [01:11:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.28, 9.27, 9.30 [01:11:41] !log [macfan@mwtask111] starting deploy of {'world': True, 'l10n': True} to all [01:11:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:11:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.50, 9.33, 8.87 [01:12:04] turns out is was cache on my end [01:12:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.63, 8.40, 7.63 [01:12:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:13:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.98, 3.63, 3.88 [01:13:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.45, 9.36, 8.94 [01:14:09] $log abort again [01:14:12] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.87, 7.53, 7.42 [01:14:21] !log abort deoloy again [01:14:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:16:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.67, 9.72, 9.60 [01:17:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.16, 3.75, 3.83 [01:17:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.04, 9.89, 9.64 [01:18:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.37, 3.85, 3.66 [01:19:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.06, 9.77, 9.27 [01:21:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.82, 9.00, 9.05 [01:22:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.46, 3.64, 3.41 [01:22:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.62, 3.74, 3.73 [01:24:13] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.39, 3.22, 3.28 [01:24:15] !log [macfan@test101] starting deploy of {'world': True} to all [01:24:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.08, 9.43, 9.38 [01:24:47] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.20, 1.71, 1.41 [01:26:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.68, 8.75, 9.16 [01:26:46] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.92, 1.49, 1.37 [01:27:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.27, 7.72, 8.26 [01:28:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.40, 3.52 [01:29:45] !log [macfan@test101] starting deploy of {'world': True} to all [01:29:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:30:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.33, 3.47 [01:31:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.26, 10.07, 9.44 [01:31:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.90, 9.79, 9.50 [01:32:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.26, 3.75, 3.43 [01:33:14] !log [macfan@test101] finished deploy of {'world': True} to all - SUCCESS in 209s [01:33:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.29, 9.35, 9.26 [01:33:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:33:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.57, 9.51, 9.43 [01:34:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.42, 3.51, 3.37 [01:35:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.07, 10.26, 9.72 [01:35:58] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.86, 3.33, 3.33 [01:36:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.90, 8.44, 8.39 [01:36:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.07, 9.24, 9.14 [01:36:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.43, 3.12, 3.39 [01:37:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.93, 9.32, 9.44 [01:38:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.35, 8.34, 8.36 [01:38:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.51, 9.67, 9.33 [01:40:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.37, 3.62, 3.53 [01:41:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.51, 9.81, 9.57 [01:42:30] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.16, 9.24, 8.74 [01:42:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.27, 3.37, 3.44 [01:43:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.06, 3.38, 3.89 [01:43:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.99, 8.77, 9.22 [01:44:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.53, 9.18, 8.77 [01:46:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.08, 2.98, 3.29 [01:48:17] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.55, 8.10, 8.46 [01:49:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.19, 8.57, 8.76 [01:51:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.19, 4.53, 4.13 [01:51:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.28, 8.36, 8.64 [01:52:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.98, 8.95, 8.70 [01:52:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.45, 7.79, 8.40 [01:53:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.63, 3.74, 3.88 [01:55:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.88, 3.91 [01:55:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.51, 8.77, 8.69 [01:57:04] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.04, 3.11, 3.62 [01:57:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.33, 8.60, 8.65 [01:57:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:57:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.56, 8.85, 8.79 [01:58:16] RECOVERY - test101 Disk Space on test101 is OK: DISK OK - free space: / 2359 MB (13% inode=57%); [01:58:30] !log [macfan@test101] starting deploy of {'world': True} to all [01:58:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:00:53] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1332 bytes in 0.025 second response time [02:01:02] ^me [02:01:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.85, 3.60, 3.67 [02:01:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.33, 8.79, 8.78 [02:01:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.59, 7.99, 8.36 [02:02:16] !log [macfan@test101] finished deploy of {'world': True} to all - SUCCESS in 226s [02:02:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:02:54] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 52154 bytes in 1.600 second response time [02:03:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.73, 3.24, 3.53 [02:03:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.68, 7.83, 8.38 [02:03:36] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.80, 9.43, 8.74 [02:05:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.28, 8.84, 8.63 [02:05:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.61, 9.20, 8.90 [02:06:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:07:02] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.91, 3.06, 3.40 [02:07:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.70, 8.41, 8.49 [02:07:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.98, 9.00, 8.86 [02:11:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.68, 7.70, 8.33 [02:17:11] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.82, 8.47, 8.33 [02:18:22] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.52, 4.37, 2.57 [02:19:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.46, 9.46, 8.70 [02:20:14] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [02:20:14] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:21:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.51, 9.35, 8.75 [02:21:23] !log [macfan@test101] starting deploy of {'world': True} to all [02:21:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:22:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.27, 8.65, 7.43 [02:22:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.58, 9.62, 8.75 [02:23:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.29, 9.89, 9.07 [02:24:12] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.70, 8.25, 7.44 [02:24:21] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.38, 9.09, 8.66 [02:24:55] !log [macfan@test101] finished deploy of {'world': True} to all - SUCCESS in 212s [02:25:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:26:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.58, 9.13, 8.68 [02:26:58] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.80, 3.08, 2.77 [02:27:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.58, 9.40, 8.98 [02:27:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 4.07, 3.58 [02:28:56] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.26, 2.79, 2.70 [02:29:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.34, 9.36, 9.02 [02:30:06] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.33, 1.76, 2.00 [02:30:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.81, 8.42, 8.50 [02:33:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.99, 3.74 [02:33:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.00, 8.13, 8.48 [02:34:54] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.01, 9.38, 8.92 [02:35:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.73, 9.56, 9.08 [02:35:57] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.57, 1.20, 1.68 [02:36:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.90, 8.98, 8.84 [02:37:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.69, 8.59, 8.80 [02:39:52] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.60, 3.00, 3.36 [02:43:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.27, 9.58, 8.98 [02:45:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.72, 9.50, 9.02 [02:46:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:46:42] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.29, 9.80, 9.28 [02:47:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.69, 8.76, 8.37 [02:47:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.46, 8.94, 8.76 [02:49:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.27, 9.20, 8.57 [02:49:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.76, 8.97, 8.80 [02:49:47] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [02:50:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:50:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.10, 9.69, 9.42 [02:51:18] !log [macfan@mwtask111] starting deploy of {'world': True, 'l10n': True} to all [02:51:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.19, 8.77, 8.49 [02:51:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:51:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.97, 9.43, 8.97 [02:51:40] * MacFan4000 crosses his fingers that nothing will break [02:52:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.34, 3.97, 3.68 [02:53:19] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.14, 8.32, 8.36 [02:53:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.45, 9.54, 9.08 [02:54:31] [02puppet] 07MacFan4000 opened pull request 03#2459: fix path to logsalmsg - 13https://github.com/miraheze/puppet/pull/2459 [02:55:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.20, 9.58, 9.14 [02:56:23] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.45, 1.85, 1.60 [02:57:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.84, 9.68, 9.23 [02:57:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.37, 7.75, 8.35 [02:58:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.60, 1.45, 1.48 [02:58:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.11, 3.52, 3.50 [02:58:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.77, 3.24, 2.94 [02:59:22] !log reset composer on mwtask111 [02:59:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:00:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.92, 3.44, 3.49 [03:00:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.23, 2.78, 2.81 [03:02:27] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.20, 7.73, 8.49 [03:02:40] !log [macfan@mwtask111] starting deploy of {'world': True, 'l10n': True} to all [03:02:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:04:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.57, 4.35, 3.81 [03:05:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.64, 10.05, 9.36 [03:06:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.08, 5.05, 3.71 [03:07:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.51, 9.54, 9.24 [03:07:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.87, 8.13, 8.06 [03:08:22] !log [macfan@mwtask111] starting deploy of {'world': True, 'l10n': True} to all [03:08:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:08:55] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 2.54, 4.37, 3.70 [03:09:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:09:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.92, 7.73, 7.93 [03:10:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.88, 8.34, 8.18 [03:10:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.70, 3.81, 3.52 [03:12:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.27, 8.84, 8.37 [03:12:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.22, 3.49, 3.52 [03:14:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.87, 8.74, 8.38 [03:15:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.89, 7.71, 8.48 [03:16:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.40, 8.33, 8.28 [03:18:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.34, 2.94, 3.22 [03:19:21] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [03:20:08] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.34, 9.23, 8.58 [03:22:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.36, 8.70, 8.48 [03:22:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.70, 3.61 [03:24:05] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.71, 7.73, 8.16 [03:24:48] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.26, 2.19, 1.50 [03:24:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.85, 4.38, 3.69 [03:27:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.24, 8.79, 8.39 [03:28:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.20, 1.75, 1.49 [03:29:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.56, 8.01, 8.13 [03:30:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.37, 3.43, 3.65 [03:30:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.21, 3.57, 3.64 [03:31:45] !log [macfan@mwtask111] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 1403s [03:31:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:32:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.79, 8.01, 7.77 [03:32:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.10, 1.50, 1.46 [03:34:40] MacFan4000: around? [03:34:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.06, 7.72, 7.70 [03:34:45] Meta returns a 500 [03:34:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.69, 3.85, 3.75 [03:34:51] !log remove Babel on mw* and rerun rsync manually [03:34:58] i know fixing now [03:34:59] A user reported it and I can reproduce [03:35:01] Thanks [03:35:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:36:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.78, 4.03, 3.74 [03:38:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.30, 3.61, 3.73 [03:38:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.29, 3.40, 3.55 [03:41:11] !log [macfan@mwtask111] starting deploy of {'world': True} to all [03:41:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:44:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.78, 3.69 [03:46:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.27, 4.55, 3.92 [03:47:50] !log [macfan@mwtask111] finished deploy of {'world': True} to all - SUCCESS in 399s [03:48:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:51:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.20, 8.65, 8.28 [03:52:26] SRE, I can reproduce as well. Any idea on cause? [03:52:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.98, 3.71, 3.79 [03:52:54] It's still being fixed at this moment it seems [03:53:17] Agent, yeah, entire farm is possibly down. 500 internal errors on at least 5-10 wikis [03:53:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.74, 9.42, 8.61 [03:53:44] !log [macfan@mwtask111] starting deploy of {'world': True} to all [03:53:51] dmehus: per -feed, some edits are going through so that's interesting [03:53:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.27, 8.37, 7.73 [03:53:59] `terribletvshowswiki`, `metawiki`, `thebestmusicandsongswikiawiki`, `theworstmusicandsongswikiawiki` etc [03:54:13] Agent, yeah, probably not *entire* farm [03:54:17] but a good chunk likely [03:55:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.88, 8.39, 8.33 [03:55:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.24, 7.84, 7.61 [03:58:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.61, 4.59, 4.10 [03:58:58] PHP Fatal Error: Cannot declare class Babel, because the name is already in use [03:59:03] thats the error [03:59:26] i was able to fix it on test101 but doing the same thing on mw* isn;t working [03:59:37] it doesn't use composer (Babel) [04:00:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.67, 3.78, 3.85 [04:01:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.85, 8.40, 7.83 [04:04:14] it doesn't help me that there isn't anybody with access around at this hour [04:04:31] (besides myself that is) [04:04:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.70, 4.16, 3.94 [04:05:50] MacFan4000, yeah... could we potentially revert and disable Babel temporarily, or would that make things worse? [04:06:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.65, 3.54, 3.74 [04:07:17] no it's not the extension specifically [04:07:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.09, 7.83, 7.81 [04:08:16] MacFan4000, oh [04:08:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.85, 3.42, 4.00 [04:09:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:11:37] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.14, 8.95, 8.34 [04:12:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.97, 2.90, 3.40 [04:13:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:14:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 3.89, 3.98 [04:15:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.57, 9.44, 8.60 [04:17:23] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.89, 8.24, 8.28 [04:18:35] urgh, sorry everybody dont think i going to be able to fix this myself, i think we'll have to wait until some of the otger sre members around [04:19:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.95, 3.34, 3.45 [04:20:17] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.16, 8.88, 8.37 [04:20:24] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [04:20:24] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:21:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.93, 7.76, 7.28 [04:21:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.48, 8.44, 7.51 [04:22:59] MacFan4000, no problem :) [04:23:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.94, 8.51, 7.67 [04:23:52] https://phabricator.miraheze.org/T9011 [04:23:53] [url] ⚓ T9011 500 errors on wikis | phabricator.miraheze.org [04:26:11] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.16, 8.37, 8.37 [04:27:04] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.59, 7.81, 7.54 [04:27:05] PROBLEM - wiki.beergeeks.co.il - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/l [04:27:05] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wiki.beergeeks.co.il. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:27:11] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.03, 8.40, 7.86 [04:28:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:28:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.13, 3.62, 3.99 [04:29:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.62, 3.06, 3.33 [04:30:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.63, 3.69, 3.95 [04:32:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.01, 3.41, 3.82 [04:33:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.91, 8.78, 8.05 [04:35:56] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.25, 8.31, 7.98 [04:37:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:38:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.40, 3.03, 3.88 [04:38:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.06, 8.91, 8.35 [04:40:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.99, 8.84, 8.39 [04:42:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.48, 7.72, 8.04 [04:45:48] .op [04:45:48] Attempting to OP... [04:46:21] [02mw-config] 07ugochimobi commented on pull request 03#4533: WBQuality constraint configs for gratisdatawiki - 13https://github.com/miraheze/mw-config/pull/4533#issuecomment-1085418579 [04:47:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:48:02] [02mw-config] 07ugochimobi commented on pull request 03#4537: T8874: Add the otherProjectsLink on gratispaideiawiki - 13https://github.com/miraheze/mw-config/pull/4537#issuecomment-1085419287 [04:48:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.50, 2.70, 3.32 [04:50:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.42, 2.97, 3.29 [04:52:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.34, 8.35, 8.08 [04:53:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:58:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:02:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.49, 9.04, 8.53 [05:04:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.97, 8.86, 8.53 [05:07:06] PROBLEM - housing.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/pyth [05:07:06] -packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query housing.wiki. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:08:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.02, 8.07, 8.29 [05:12:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:13:44] [02mw-config] 07ugochimobi commented on pull request 03#4537: T8874: Add the otherProjectsLink on gratispaideiawiki - 13https://github.com/miraheze/mw-config/pull/4537#issuecomment-1085431560 [05:19:31] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [05:26:53] RECOVERY - wiki.beergeeks.co.il - reverse DNS on sslhost is OK: SSL OK - wiki.beergeeks.co.il reverse DNS resolves to cp21.miraheze.org - CNAME OK [05:27:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:36:46] RECOVERY - housing.wiki - reverse DNS on sslhost is OK: SSL OK - housing.wiki reverse DNS resolves to cp30.miraheze.org - CNAME FLAT [05:48:25] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/8ed697a0bf4a [05:48:27] [02miraheze/mw-config] 07Reception123 038ed697a - temporarily disable babel until T9011 is figured out [05:48:28] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config [05:48:30] [02mw-config] 07Reception123 opened pull request 03#4542: temporarily disable babel until T9011 is figured out - 13https://github.com/miraheze/mw-config/pull/4542 [05:49:31] miraheze/mw-config - Reception123 the build passed. [05:49:40] [02mw-config] 07Reception123 closed pull request 03#4542: temporarily disable babel until T9011 is figured out - 13https://github.com/miraheze/mw-config/pull/4542 [05:49:41] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a05b79dfd731...8f2b351fe41e [05:49:43] [02miraheze/mw-config] 07Reception123 038f2b351 - temporarily disable babel until T9011 is figured out (#4542) [05:49:44] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [05:49:46] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config [05:49:52] !log [reception@mwtask111] starting deploy of {'config': True} to all [05:50:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:50:07] !log [reception@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 14s [05:50:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:50:39] miraheze/mw-config - Reception123 the build passed. [05:51:44] !log [reception@mwtask111] starting deploy of {'l10n': True} to all [05:51:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:52:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:53:15] !log [@mwtask111] starting deploy of {'config': True} to all [05:53:22] !log [reception@mwtask111] finished deploy of {'l10n': True} to all - SUCCESS in 98s [05:53:23] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 6s [05:54:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.77, 1.65, 1.23 [05:56:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.06, 1.43, 1.20 [05:57:06] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.02, 2.30, 1.28 [05:57:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:57:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.60, 3.92, 3.08 [05:58:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 3.43, 2.94 [05:59:17] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.82, 3.77, 2.96 [06:01:04] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.00, 1.87, 1.35 [06:01:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.96, 3.57, 2.98 [06:02:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.66, 3.17 [06:03:04] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.55, 1.40, 1.24 [06:03:13] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.92, 3.30, 2.95 [06:06:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.88, 3.36 [06:07:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.58, 3.80, 3.21 [06:08:33] PROBLEM - housing.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/pyth [06:08:34] -packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query housing.wiki. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:08:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.63, 3.33 [06:08:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.87, 1.71, 1.43 [06:10:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:10:48] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.57, 1.97, 1.56 [06:11:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.32, 3.29, 3.16 [06:11:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.53, 3.68, 3.73 [06:12:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.43, 1.77, 1.54 [06:14:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.16, 1.58, 1.50 [06:15:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:17:46] !log [@test101] starting deploy of {'config': True} to all [06:17:47] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [06:17:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.31, 8.68, 7.89 [06:18:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.20, 3.93, 3.55 [06:18:49] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.01, 1.76, 1.58 [06:19:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.53, 7.81, 7.67 [06:19:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.62, 3.91, 3.76 [06:20:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.61, 3.48 [06:21:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.70, 9.44, 8.25 [06:21:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:22:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.57, 3.27, 3.37 [06:23:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 3.68, 3.70 [06:25:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.12, 9.12, 8.29 [06:25:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.78, 3.91, 3.76 [06:27:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.28, 3.59, 3.66 [06:31:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.38, 8.33, 8.24 [06:31:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.31, 3.02, 3.40 [06:33:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.77, 8.29, 8.20 [06:34:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.87, 8.70, 8.23 [06:35:38] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.95, 10.11, 8.95 [06:36:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.76, 4.04, 3.57 [06:36:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.97, 9.21, 8.46 [06:37:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.08, 9.65, 8.93 [06:38:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.65, 3.56, 3.46 [06:38:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.69, 9.31, 8.58 [06:38:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.69, 1.96, 2.00 [06:39:41] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/p [06:39:42] ist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query seamly.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:40:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.35, 9.63, 8.79 [06:40:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.79, 4.03, 3.64 [06:40:48] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.39, 2.11, 2.04 [06:41:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:41:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.71, 8.02, 8.42 [06:42:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.43, 9.35, 8.78 [06:42:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.76, 3.59 [06:42:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.59, 1.95, 2.00 [06:43:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 3.77, 3.47 [06:44:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.14, 4.37, 3.84 [06:44:48] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.24, 2.10, 2.04 [06:45:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.82, 3.54 [06:46:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:46:45] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.43, 7.94, 8.33 [06:47:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.04, 8.56, 8.50 [06:48:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.94, 3.95, 3.77 [06:49:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.79, 3.57 [06:50:08] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [06:50:08] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:51:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.82, 8.63, 8.20 [06:51:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.18, 3.60, 3.53 [06:53:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.33, 8.01, 8.04 [06:53:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.15, 9.17, 8.75 [06:53:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 3.64, 3.54 [06:55:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.20, 3.09, 3.34 [06:56:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:56:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.13, 9.93, 8.90 [06:56:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.78, 2.83, 3.34 [06:57:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.83, 9.06, 8.77 [06:58:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.67, 9.57, 8.91 [06:59:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:01:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.12, 7.98, 8.44 [07:04:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:06:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 3.08, 3.11 [07:08:42] RECOVERY - wiki.seamly.net - reverse DNS on sslhost is OK: SSL OK - wiki.seamly.net reverse DNS resolves to cp20.miraheze.org - CNAME OK [07:08:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 3.24, 3.15 [07:10:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.78, 2.90, 3.03 [07:12:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.88, 7.89, 8.43 [07:25:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:28:50] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.25, 1.70, 2.00 [07:30:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:31:50] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(h [07:31:50] 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406602621078491 seconds [07:36:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.72, 1.18, 1.63 [07:37:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.75, 8.52, 8.05 [07:37:50] RECOVERY - housing.wiki - reverse DNS on sslhost is OK: SSL OK - housing.wiki reverse DNS resolves to cp30.miraheze.org - CNAME FLAT [07:39:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.76, 8.03, 7.91 [07:42:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:43:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.36, 8.97, 8.35 [07:45:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.98, 8.24, 8.16 [07:45:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.85, 5.12, 3.84 [07:53:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.21, 3.34, 3.51 [07:55:30] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.29, 2.93, 3.33 [07:56:09] PROBLEM - ref.undu.st - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ref.undu.st could not be found [07:57:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:01:31] RECOVERY - wiki.zymonic.com - reverse DNS on sslhost is OK: SSL OK - wiki.zymonic.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [08:18:49] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [09:17:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.31, 8.67, 7.31 [09:19:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.93, 8.63, 7.45 [09:24:49] [02puppet] 07paladox closed pull request 03#2459: fix path to logsalmsg - 13https://github.com/miraheze/puppet/pull/2459 [09:24:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7651a617883a...2b86e3773774 [09:24:52] [02miraheze/puppet] 07MacFan4000 032b86e37 - fix path to logsalmsg (#2459) [09:25:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.38, 7.78, 7.51 [09:29:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:34:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:28:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.59, 3.21, 2.62 [10:30:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.23, 2.75, 2.51 [10:32:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:37:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:50:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:00:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:04:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.84, 8.39, 7.90 [11:06:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.04, 7.99, 7.81 [11:18:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:19:00] PROBLEM - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.rosestulipsandliberty.com' expires in 15 day(s) (Sun 17 Apr 2022 11:15:49 GMT +0000). [11:22:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.93, 8.30, 7.72 [11:24:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.82, 7.98, 7.67 [11:25:20] PROBLEM - otogebase.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'otogebase.com' expires in 15 day(s) (Sun 17 Apr 2022 11:19:40 GMT +0000). [11:25:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/454869821d26...75dac6ac2426 [11:25:22] [02miraheze/ssl] 07MirahezeSSLBot 0375dac6a - Bot: Update SSL cert for wiki.rosestulipsandliberty.com [11:27:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/75dac6ac2426...cdb6992fa80b [11:27:08] [02miraheze/ssl] 07MirahezeSSLBot 03cdb6992 - Bot: Update SSL cert for otogebase.com [11:28:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:30:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.85, 8.89, 8.16 [11:31:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.15, 8.65, 8.16 [11:33:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.88, 8.24, 8.08 [11:34:47] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.16, 9.35, 8.47 [11:36:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.13, 9.31, 8.57 [11:38:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.52, 8.48, 8.35 [11:40:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:42:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.40, 9.34, 8.72 [11:44:29] PROBLEM - mw102 Puppet on mw102 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/logrotate.d/nginx] [11:44:35] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/logrotate.d/nginx] [11:44:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.60, 8.52, 8.49 [11:48:37] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.70, 9.37, 8.81 [11:49:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.74, 3.23 [11:50:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.75, 9.26, 8.85 [11:50:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.95, 8.71, 8.03 [11:55:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.22, 3.38, 3.29 [11:58:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.13, 9.71, 9.21 [11:58:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 2.97, 2.49 [12:00:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.04, 9.06, 9.03 [12:00:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.49, 2.85, 2.51 [12:02:29] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.74, 9.44, 9.15 [12:04:05] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.24, 9.96, 8.66 [12:04:28] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.96, 9.26, 9.12 [12:04:35] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.10, 9.62, 8.88 [12:06:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.54, 10.63, 9.65 [12:06:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.03, 9.16, 8.80 [12:08:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.94, 9.75, 9.43 [12:08:31] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.65, 9.53, 8.97 [12:10:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.47, 4.11, 3.60 [12:10:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.11, 9.11, 8.89 [12:12:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.60, 3.47 [12:12:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.50, 9.63, 9.40 [12:12:33] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:12:46] RECOVERY - mw102 Puppet on mw102 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:13:32] hi [12:14:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.09, 3.14, 3.33 [12:14:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.55, 8.97, 9.20 [12:17:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.61, 9.43, 9.49 [12:18:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.59, 9.86, 9.24 [12:19:59] hello [12:20:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.93, 9.87, 9.35 [12:21:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.10, 9.21, 9.35 [12:21:43] hi [12:22:01] Guest51 Yes? [12:22:04] .op [12:22:04] Attempting to OP... [12:22:09] wait [12:22:15] come on [12:22:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.31, 9.15, 9.20 [12:24:00] RECOVERY - otogebase.com - LetsEncrypt on sslhost is OK: OK - Certificate 'otogebase.com' will expire on Thu 30 Jun 2022 10:27:01 GMT +0000. [12:24:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.54, 8.89, 9.11 [12:25:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.95, 9.38, 9.46 [12:26:14] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.66, 9.87, 9.43 [12:28:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.53, 9.35, 9.29 [12:34:06] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.28, 9.66, 9.43 [12:34:09] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.41, 9.38, 9.32 [12:36:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.92, 9.49, 9.40 [12:38:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.06, 9.36, 9.42 [12:38:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.08, 8.82, 8.91 [12:39:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.52, 9.08, 7.85 [12:40:01] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.04, 9.90, 9.54 [12:40:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.05, 9.92, 9.61 [12:41:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.14, 8.44, 7.78 [12:42:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.19, 9.63, 9.55 [12:42:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.43, 9.31, 9.12 [12:44:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.60, 9.87, 9.33 [12:45:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:46:46] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.99, 9.74, 9.37 [12:51:13] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.58, 3.10 [12:51:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:51:58] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.43, 9.87, 9.56 [12:51:59] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.59, 3.57, 3.45 [12:52:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.24, 3.69, 3.19 [12:52:48] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.07, 1.72, 1.50 [12:53:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.69, 3.31, 3.06 [12:53:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.67, 9.49, 9.84 [12:54:05] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.38, 3.22, 3.08 [12:54:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.56, 1.72, 1.54 [12:55:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.00, 9.90, 9.67 [12:55:57] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.78, 3.24, 3.35 [12:56:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.51, 9.86, 9.34 [12:57:54] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.17, 10.47, 9.92 [12:58:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.62, 9.28, 9.19 [12:59:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.47, 9.82, 9.75 [13:00:48] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.09, 1.96, 1.70 [13:01:28] PROBLEM - test101 Puppet on test101 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 minutes ago with 0 failures [13:01:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.42, 10.88, 10.15 [13:02:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.61, 1.82, 1.68 [13:03:38] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.18, 9.57, 9.69 [13:04:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.43, 9.58, 9.24 [13:05:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.20, 8.87, 9.44 [13:06:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.73, 9.04, 9.07 [13:06:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.37, 1.67, 1.66 [13:08:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 3.16, 2.89 [13:09:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.11, 9.66, 9.97 [13:09:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.78, 3.18, 3.05 [13:10:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.18, 9.50, 9.23 [13:10:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.46, 2.82, 2.79 [13:11:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.36, 9.79, 9.97 [13:12:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.90, 8.17, 7.80 [13:13:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.18, 4.44, 3.52 [13:14:34] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.89, 7.60, 7.63 [13:15:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.57, 9.29, 9.81 [13:16:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.92, 3.71, 3.18 [13:18:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.10, 3.74, 3.24 [13:19:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.43, 9.76, 9.45 [13:20:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.17, 3.32, 3.16 [13:21:39] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.53, 9.29, 9.56 [13:21:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.33, 3.91, 3.80 [13:24:03] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [13:24:03] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:24:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.68, 9.76, 9.85 [13:25:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.23, 9.11, 9.50 [13:26:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.20, 9.93, 9.90 [13:27:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.17, 3.79, 3.72 [13:28:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.94, 9.55, 9.76 [13:29:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.45, 9.49, 9.49 [13:30:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.50, 3.50, 3.29 [13:31:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.05, 3.66, 3.72 [13:32:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.80, 10.55, 10.08 [13:33:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.06, 9.59, 9.56 [13:34:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.80, 3.25, 3.24 [13:35:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.39, 11.07, 10.11 [13:37:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.10, 3.80, 3.65 [13:39:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.33, 9.56, 9.71 [13:40:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.86, 3.52 [13:41:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.51, 10.41, 10.01 [13:43:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.11, 9.07, 9.56 [13:43:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.79, 3.93, 3.70 [13:44:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.46, 9.71, 8.52 [13:44:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 4.10, 3.69 [13:46:21] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.65, 9.39, 8.53 [13:46:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.79, 3.63 [13:49:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.36, 2.86, 3.32 [13:50:16] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.09, 8.15, 8.21 [13:50:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.12, 3.59, 3.57 [13:51:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.66, 10.08, 9.71 [13:52:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.45, 3.52 [13:53:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.51, 3.36, 3.06 [13:53:37] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [13:54:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.59, 3.06, 3.36 [13:56:58] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.22, 3.40, 3.14 [13:59:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.85, 9.21, 9.57 [14:02:51] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03revert-4542-Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/3a242b62772d [14:02:52] [02miraheze/mw-config] 07RhinosF1 033a242b6 - Revert "temporarily disable babel until T9011 is figured out (#4542)" [14:02:54] [02mw-config] 07RhinosF1 created branch 03revert-4542-Reception123-patch-1 - 13https://github.com/miraheze/mw-config [14:02:55] [02mw-config] 07RhinosF1 opened pull request 03#4543: Revert "temporarily disable babel until T9011 is figured out" - 13https://github.com/miraheze/mw-config/pull/4543 [14:03:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.92, 10.23, 9.90 [14:05:34] miraheze/mw-config - RhinosF1 the build passed. [14:15:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.27, 9.00, 9.97 [14:16:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.07, 3.85, 3.31 [14:16:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.86, 3.44, 3.11 [14:17:42] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.97, 8.50, 8.15 [14:18:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.29, 3.66, 3.23 [14:19:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.11, 10.22, 10.21 [14:20:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.70, 3.36 [14:20:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.99, 2.99, 3.03 [14:21:09] !log [rhinos@test101] starting deploy of {'config': True} to all [14:21:10] !log [rhinos@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [14:21:41] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.96, 9.47, 8.60 [14:22:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.24, 3.07, 3.16 [14:23:41] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.31, 8.43, 8.34 [14:26:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.55, 8.42, 8.04 [14:30:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.44, 7.94, 7.93 [14:31:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.21, 8.60, 8.36 [14:34:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.68, 8.53, 8.18 [14:35:41] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.60, 8.21, 8.31 [14:36:34] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.09, 7.64, 7.90 [14:37:11] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:37:25] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:37:41] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:38:04] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:38:12] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:38:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.82, 3.70, 3.11 [14:39:07] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.700 second response time [14:39:23] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.318 second response time [14:39:36] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.664 second response time [14:40:02] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.350 second response time [14:40:06] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.016 second response time [14:43:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.00, 9.06, 8.37 [14:44:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.47, 3.17 [14:44:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.90, 3.64, 3.29 [14:46:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.95, 3.38, 3.18 [14:46:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.90, 3.26, 3.19 [14:50:37] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.43, 8.45, 8.40 [14:54:40] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [14:54:40] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:54:53] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 274 bytes in 0.028 second response time [14:55:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.15, 8.91, 9.79 [14:56:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.49, 3.27 [14:57:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.16, 8.60, 8.47 [14:57:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.35, 9.13, 9.97 [14:57:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.25, 9.67, 9.95 [14:59:11] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.92, 8.36, 8.40 [14:59:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.16, 9.66, 9.93 [15:00:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.37, 8.42, 7.98 [15:00:50] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.41, 4.20, 3.62 [15:01:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.81, 10.60, 10.34 [15:02:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.86, 3.56 [15:02:59] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.04, 8.78, 8.57 [15:03:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.80, 9.78, 9.87 [15:04:29] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.87, 7.69, 7.83 [15:04:53] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.18, 8.18, 8.37 [15:06:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.51, 3.13, 3.33 [15:07:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.64, 9.62, 9.82 [15:08:21] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.54, 8.34, 8.00 [15:08:54] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 52002 bytes in 0.905 second response time [15:09:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.14, 10.03, 9.93 [15:12:11] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.85, 9.82, 8.63 [15:12:25] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.93, 2.93, 3.87 [15:13:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.40, 9.38, 9.90 [15:14:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.62, 3.77, 4.08 [15:16:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.09, 9.12, 8.67 [15:16:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.08, 3.69, 3.33 [15:17:21] !log [@test101] starting deploy of {'config': True} to all [15:17:22] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [15:17:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.90, 10.15, 10.05 [15:17:28] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:17:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.38, 9.07, 9.68 [15:18:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.82, 9.54, 9.99 [15:18:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.22, 3.38, 3.25 [15:21:28] PROBLEM - test101 Puppet on test101 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [15:21:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.61, 7.69, 8.23 [15:22:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.44, 3.88 [15:23:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.98, 9.90, 9.81 [15:23:39] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: APT CRITICAL: 60 packages available for upgrade (2 critical updates). [15:24:13] PROBLEM - es111 APT on es111 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (1 critical updates). [15:24:13] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [15:24:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.54, 9.65, 9.86 [15:25:09] PROBLEM - puppet111 APT on puppet111 is CRITICAL: APT CRITICAL: 46 packages available for upgrade (1 critical updates). [15:25:32] PROBLEM - cp20 APT on cp20 is CRITICAL: APT CRITICAL: 38 packages available for upgrade (1 critical updates). [15:27:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.69, 9.23, 9.56 [15:28:15] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 41 packages available for upgrade (1 critical updates). [15:28:24] PROBLEM - cp21 APT on cp21 is CRITICAL: APT CRITICAL: 38 packages available for upgrade (1 critical updates). [15:28:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.03, 9.91, 9.94 [15:30:00] PROBLEM - graylog121 APT on graylog121 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (1 critical updates). [15:30:15] PROBLEM - gluster121 APT on gluster121 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (1 critical updates). [15:30:18] PROBLEM - gluster111 APT on gluster111 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (1 critical updates). [15:30:40] PROBLEM - mon111 APT on mon111 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (1 critical updates). [15:31:46] PROBLEM - mw101 APT on mw101 is CRITICAL: APT CRITICAL: 56 packages available for upgrade (2 critical updates). [15:32:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.61, 3.04, 2.84 [15:32:22] !log [rhinos@test101] starting deploy of {'world': True} to all [15:32:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.39, 4.34, 3.46 [15:33:10] !log [rhinos@test101] starting deploy of {'world': True, 'ignoretime': True} to all [15:33:10] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.47, 8.65, 7.95 [15:33:15] !log [rhinos@test101] starting deploy of {'world': True, 'force': True, 'ignoretime': True} to all [15:33:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.81, 9.59, 9.99 [15:33:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.95, 9.57, 9.50 [15:34:04] PROBLEM - mem121 APT on mem121 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (1 critical updates). [15:34:08] PROBLEM - db112 APT on db112 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (1 critical updates). [15:34:11] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.05, 2.74, 2.76 [15:34:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.28, 8.72, 8.29 [15:34:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.10, 10.32, 10.06 [15:35:51] PROBLEM - bast101 APT on bast101 is CRITICAL: APT CRITICAL: 41 packages available for upgrade (1 critical updates). [15:36:05] PROBLEM - mw122 APT on mw122 is CRITICAL: APT CRITICAL: 56 packages available for upgrade (2 critical updates). [15:36:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.85, 2.80, 3.33 [15:36:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.01, 3.32, 3.27 [15:36:53] PROBLEM - mw111 APT on mw111 is CRITICAL: APT CRITICAL: 56 packages available for upgrade (2 critical updates). [15:37:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.53, 9.45, 9.50 [15:38:12] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.66, 8.38, 8.27 [15:39:49] PROBLEM - phab121 APT on phab121 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (1 critical updates). [15:40:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.17, 3.72, 3.55 [15:40:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.42, 9.63, 9.85 [15:41:02] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 4.23, 2.47, 1.22 [15:41:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.02, 8.81, 9.33 [15:41:25] PROBLEM - cloud10 APT on cloud10 is CRITICAL: APT CRITICAL: 60 packages available for upgrade (1 critical updates). [15:41:26] PROBLEM - db111 APT on db111 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (1 critical updates). [15:42:06] PROBLEM - cloud12 APT on cloud12 is CRITICAL: APT CRITICAL: 60 packages available for upgrade (1 critical updates). [15:42:09] PROBLEM - mw102 APT on mw102 is CRITICAL: APT CRITICAL: 56 packages available for upgrade (2 critical updates). [15:42:11] PROBLEM - bast121 APT on bast121 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (1 critical updates). [15:42:16] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.41, 3.53, 3.50 [15:43:04] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.31, 8.35, 8.21 [15:43:25] !log [rhinos@test101] finished deploy of {'world': True, 'force': True, 'ignoretime': True} to all - SUCCESS in 610s [15:43:45] PROBLEM - mw121 APT on mw121 is CRITICAL: APT CRITICAL: 56 packages available for upgrade (2 critical updates). [15:44:09] PROBLEM - ldap111 APT on ldap111 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (1 critical updates). [15:44:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.89, 4.44, 3.84 [15:44:29] PROBLEM - mail121 APT on mail121 is CRITICAL: APT CRITICAL: 49 packages available for upgrade (1 critical updates). [15:44:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.52, 9.98, 9.89 [15:45:09] CosmicAlpha: hello [15:45:27] Hello ugochimobi [15:45:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.84, 9.69, 9.52 [15:45:56] CosmicAlpha: Welcome here :) [15:46:01] PROBLEM - es121 APT on es121 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (1 critical updates). [15:46:12] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.88, 9.09, 8.55 [15:46:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.59, 3.43 [15:46:55] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 786 bytes in 0.407 second response time [15:47:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.35, 9.18, 8.55 [15:47:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 0.22, 1.45, 1.26 [15:47:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.79, 9.97, 9.72 [15:47:25] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.95, 2.07, 1.56 [15:47:31] PROBLEM - gluster101 APT on gluster101 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (1 critical updates). [15:47:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.82, 9.37, 9.43 [15:48:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.86, 8.58, 8.43 [15:48:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.68, 3.91, 3.76 [15:48:54] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 52002 bytes in 0.828 second response time [15:49:00] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.69, 8.47, 8.38 [15:49:14] PROBLEM - jobchron121 APT on jobchron121 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (1 critical updates). [15:49:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.25, 11.24, 10.21 [15:49:24] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.72, 1.57, 1.44 [15:49:27] PROBLEM - es101 APT on es101 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (1 critical updates). [15:49:51] PROBLEM - test101 APT on test101 is CRITICAL: APT CRITICAL: 61 packages available for upgrade (2 critical updates). [15:50:12] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.16, 8.16, 8.30 [15:50:18] PROBLEM - cp31 APT on cp31 is CRITICAL: APT CRITICAL: 38 packages available for upgrade (1 critical updates). [15:50:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.54, 9.90, 9.94 [15:50:49] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (1 critical updates). [15:51:14] PROBLEM - mw112 APT on mw112 is CRITICAL: APT CRITICAL: 56 packages available for upgrade (2 critical updates). [15:51:55] PROBLEM - db121 APT on db121 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (1 critical updates). [15:52:11] PROBLEM - prometheus101 APT on prometheus101 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (1 critical updates). [15:52:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.30, 3.36, 3.39 [15:52:47] PROBLEM - matomo101 APT on matomo101 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (1 critical updates). [15:53:05] PROBLEM - db101 APT on db101 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (1 critical updates). [15:53:08] PROBLEM - mem101 APT on mem101 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (1 critical updates). [15:53:18] PROBLEM - cloud11 APT on cloud11 is CRITICAL: APT CRITICAL: 60 packages available for upgrade (1 critical updates). [15:53:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.10, 9.54, 9.79 [15:53:42] !log security update for zlib1g [15:53:51] RECOVERY - bast101 APT on bast101 is OK: APT OK: 40 packages available for upgrade (0 critical updates). [15:53:56] RECOVERY - db121 APT on db121 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [15:54:00] RECOVERY - graylog121 APT on graylog121 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [15:54:05] RECOVERY - cloud12 APT on cloud12 is OK: APT OK: 59 packages available for upgrade (0 critical updates). [15:54:08] RECOVERY - ldap111 APT on ldap111 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [15:54:09] RECOVERY - es121 APT on es121 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [15:54:11] RECOVERY - bast121 APT on bast121 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [15:54:11] RECOVERY - es111 APT on es111 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [15:54:12] RECOVERY - db112 APT on db112 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [15:54:12] RECOVERY - prometheus101 APT on prometheus101 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [15:54:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.85, 5.03, 4.16 [15:54:14] RECOVERY - gluster121 APT on gluster121 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [15:54:15] RECOVERY - ns1 APT on ns1 is OK: APT OK: 40 packages available for upgrade (0 critical updates). [15:54:17] RECOVERY - cp31 APT on cp31 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [15:54:17] RECOVERY - gluster111 APT on gluster111 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [15:54:22] RECOVERY - cp21 APT on cp21 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [15:54:39] RECOVERY - mon111 APT on mon111 is OK: APT OK: 47 packages available for upgrade (0 critical updates). [15:54:40] RECOVERY - mail121 APT on mail121 is OK: APT OK: 48 packages available for upgrade (0 critical updates). [15:54:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.03, 9.48, 9.69 [15:54:48] RECOVERY - ns2 APT on ns2 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [15:54:49] RECOVERY - matomo101 APT on matomo101 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [15:54:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.51, 3.31, 3.15 [15:54:53] RECOVERY - mw111 APT on mw111 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [15:55:06] RECOVERY - db101 APT on db101 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [15:55:11] RECOVERY - mem101 APT on mem101 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [15:55:11] RECOVERY - puppet111 APT on puppet111 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [15:55:14] RECOVERY - jobchron121 APT on jobchron121 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [15:55:16] RECOVERY - mw112 APT on mw112 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [15:55:18] RECOVERY - cloud11 APT on cloud11 is OK: APT OK: 59 packages available for upgrade (0 critical updates). [15:55:27] RECOVERY - db111 APT on db111 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [15:55:27] RECOVERY - cloud10 APT on cloud10 is OK: APT OK: 59 packages available for upgrade (0 critical updates). [15:55:27] RECOVERY - gluster101 APT on gluster101 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [15:55:29] RECOVERY - es101 APT on es101 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [15:55:29] RECOVERY - cp20 APT on cp20 is OK: APT OK: 37 packages available for upgrade (0 critical updates). [15:55:33] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [15:55:41] RECOVERY - mw121 APT on mw121 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [15:55:45] RECOVERY - mw101 APT on mw101 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [15:55:51] RECOVERY - test101 APT on test101 is OK: APT OK: 59 packages available for upgrade (0 critical updates). [15:56:02] RECOVERY - mem121 APT on mem121 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [15:56:05] RECOVERY - phab121 APT on phab121 is OK: APT OK: 43 packages available for upgrade (0 critical updates). [15:56:08] RECOVERY - mw122 APT on mw122 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [15:56:12] RECOVERY - mw102 APT on mw102 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [15:56:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.50, 9.37, 9.61 [15:56:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.58, 3.11, 3.10 [15:56:54] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.43, 8.53, 8.29 [15:57:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.04, 10.49, 10.07 [15:57:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.72, 9.54, 9.22 [15:58:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.73, 3.71, 3.82 [15:58:53] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.75, 8.16, 8.18 [15:59:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.61, 9.62, 9.80 [15:59:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.15, 9.10, 9.11 [16:00:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.12, 9.45, 9.58 [16:01:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.72, 10.10, 9.95 [16:01:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.99, 3.91, 3.62 [16:01:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.56, 10.06, 9.46 [16:02:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.42, 3.88, 3.87 [16:02:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.27, 9.57, 9.62 [16:02:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.56, 8.46, 8.31 [16:03:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.37, 9.85, 9.89 [16:03:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.44, 9.35, 9.27 [16:05:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.94, 4.54, 3.91 [16:05:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.51, 4.11, 3.56 [16:06:48] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.23, 8.31, 8.29 [16:07:25] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:46] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:09:24] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.776 second response time [16:09:42] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.523 second response time [16:10:20] !log security update for zlib1g [16:10:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.16, 9.18, 9.36 [16:10:51] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:54] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:58] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:11:10] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:11:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.24, 3.69, 3.81 [16:11:31] !log security update for zlib1g [16:11:39] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.05, 3.69, 3.62 [16:12:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.88, 8.01, 7.67 [16:12:25] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:14:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.22, 7.91, 7.69 [16:14:28] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.438 second response time [16:14:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.49, 9.68, 9.54 [16:15:06] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.842 second response time [16:15:06] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.077 second response time [16:15:08] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 9.257 second response time [16:17:30] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.774 second response time [16:17:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.01, 7.44, 8.41 [16:19:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.60, 9.65, 9.52 [16:20:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.54, 9.88, 9.68 [16:21:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.29, 9.29, 9.42 [16:22:38] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.21, 8.45, 7.90 [16:22:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.42, 9.87, 9.72 [16:23:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 3.70, 3.70 [16:23:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.37, 9.06, 8.78 [16:24:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.93, 7.49, 7.61 [16:25:24] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.54, 3.85, 3.66 [16:26:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.01, 9.92, 9.74 [16:27:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.13, 3.64, 3.68 [16:28:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.41, 9.68, 9.67 [16:29:53] !log [rhinos@test101] starting deploy of {'folders': 'w/extensions/UniversalLanguageSelector'} to all [16:29:54] !log [rhinos@test101] finished deploy of {'folders': 'w/extensions/UniversalLanguageSelector'} to all - SUCCESS in 0s [16:31:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.38, 3.38, 3.55 [16:31:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.45, 9.62, 9.34 [16:32:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.03, 10.38, 9.96 [16:32:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.47, 3.73, 3.70 [16:33:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.66, 3.73, 3.65 [16:33:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.10, 9.52, 9.31 [16:34:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.69, 3.37, 3.57 [16:34:57] !log [rhinos@test101] starting deploy of {'world': True, 'force': True, 'ignoretime': True} to all [16:35:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.18, 3.54, 3.59 [16:35:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.01, 10.06, 9.65 [16:35:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.81, 9.99, 9.54 [16:36:47] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.56, 2.76, 3.32 [16:37:11] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.48, 2.77, 3.30 [16:37:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.83, 9.88, 9.63 [16:38:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.06, 8.10, 7.69 [16:40:14] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.86, 8.00, 7.71 [16:40:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.53, 9.47, 9.83 [16:41:42] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03revert-6106-dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-189ff5b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9779f189ef47 [16:41:43] [02miraheze/mediawiki] 07RhinosF1 039779f18 - Revert "Bump extensions/UniversalLanguageSelector from `28c52ed` to `189ff5b` (#6106)" [16:41:45] [02mediawiki] 07RhinosF1 created branch 03revert-6106-dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-189ff5b - 13https://github.com/miraheze/mediawiki [16:41:46] [02mediawiki] 07RhinosF1 opened pull request 03#6144: Revert "Bump extensions/UniversalLanguageSelector from `28c52ed` to `189ff5b`" - 13https://github.com/miraheze/mediawiki/pull/6144 [16:42:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.05, 4.20, 3.74 [16:43:01] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.34, 2.10, 1.08 [16:43:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.75, 3.50, 3.44 [16:43:09] !log [rhinos@test101] finished deploy of {'world': True, 'force': True, 'ignoretime': True} to all - SUCCESS in 492s [16:43:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.85, 10.25, 9.80 [16:43:30] !log [rhinos@test101] starting deploy of {'folders': 'w/extensions/UniversalLanguageSelector'} to all [16:43:31] !log [rhinos@test101] finished deploy of {'folders': 'w/extensions/UniversalLanguageSelector'} to all - SUCCESS in 0s [16:44:13] [02mediawiki] 07RhinosF1 closed pull request 03#6144: Revert "Bump extensions/UniversalLanguageSelector from `28c52ed` to `189ff5b`" - 13https://github.com/miraheze/mediawiki/pull/6144 [16:44:14] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03revert-6106-dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-189ff5b [16:44:16] [02mediawiki] 07RhinosF1 deleted branch 03revert-6106-dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-189ff5b - 13https://github.com/miraheze/mediawiki [16:44:17] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/f52378334782...d80880efbbdb [16:44:19] [02miraheze/mediawiki] 07RhinosF1 03d80880e - Revert "Bump extensions/UniversalLanguageSelector from `28c52ed` to `189ff5b` (#6106)" (#6144) [16:44:20] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03revert-5986-dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-28c52ed [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/fceacd72464d [16:44:22] [02miraheze/mediawiki] 07RhinosF1 03fceacd7 - Revert "Bump extensions/UniversalLanguageSelector from `32ee5f5` to `28c52ed` (#5986)" [16:44:23] [02mediawiki] 07RhinosF1 created branch 03revert-5986-dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-28c52ed - 13https://github.com/miraheze/mediawiki [16:44:25] [02mediawiki] 07RhinosF1 opened pull request 03#6145: Revert "Bump extensions/UniversalLanguageSelector from `32ee5f5` to `28c52ed`" - 13https://github.com/miraheze/mediawiki/pull/6145 [16:44:30] [02mediawiki] 07RhinosF1 closed pull request 03#6145: Revert "Bump extensions/UniversalLanguageSelector from `32ee5f5` to `28c52ed`" - 13https://github.com/miraheze/mediawiki/pull/6145 [16:44:32] [02mediawiki] 07RhinosF1 deleted branch 03revert-5986-dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-28c52ed - 13https://github.com/miraheze/mediawiki [16:44:33] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03revert-5986-dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-28c52ed [16:44:35] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/d80880efbbdb...32dc7f509d41 [16:44:36] [02miraheze/mediawiki] 07RhinosF1 0332dc7f5 - Revert "Bump extensions/UniversalLanguageSelector from `32ee5f5` to `28c52ed` (#5986)" (#6145) [16:44:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.62, 9.40, 9.65 [16:45:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.63, 3.24, 3.36 [16:45:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 0.72, 1.56, 1.01 [16:45:22] !log [rhinos@mwtask111] starting deploy of {'world': True, 'l10n': True} to all [16:45:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.76, 9.96, 9.78 [16:45:29] !log [rhinos@mwtask111] starting deploy of {'world': True, 'l10n': True, 'ignoretime': True} to all [16:46:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.37, 3.43, 3.55 [16:47:29] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:47:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.64, 10.03, 9.63 [16:48:44] !log test101 is back to matching prod [16:48:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.92, 9.83, 9.84 [16:48:57] !log [@test101] starting deploy of {'config': True} to all [16:48:58] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [16:49:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.03, 10.00, 9.77 [16:49:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.98, 8.99, 9.28 [16:50:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.41, 3.06, 3.39 [16:51:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.66, 9.34, 9.56 [16:51:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:51:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.55, 9.52, 9.43 [16:52:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.56, 9.72, 9.75 [16:52:48] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.87, 2.29, 1.16 [16:54:50] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [16:54:50] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:54:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.66, 3.85, 3.46 [16:55:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.34, 10.40, 9.90 [16:56:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.22, 3.59, 3.99 [16:56:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.39, 9.58, 9.70 [16:57:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.66, 9.84, 9.74 [16:57:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.37, 9.84, 9.68 [16:58:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.74, 10.32, 9.95 [16:59:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.91, 10.49, 10.01 [16:59:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.37, 10.99, 10.14 [17:00:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.85, 3.93, 3.70 [17:03:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.73, 9.60, 9.76 [17:03:58] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.51, 2.79, 3.39 [17:04:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.08, 2.98, 3.37 [17:05:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.89, 9.54, 9.84 [17:05:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.16, 9.65, 9.74 [17:07:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.15, 10.05, 10.00 [17:07:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.41, 9.13, 9.53 [17:09:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.93, 9.67, 9.85 [17:11:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.25, 10.52, 10.13 [17:13:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.00, 9.79, 9.90 [17:15:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.01, 9.76, 9.87 [17:17:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.24, 8.94, 9.56 [17:18:41] !log [@test101] starting deploy of {'config': True} to all [17:18:42] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [17:18:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.70, 1.58, 1.95 [17:19:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.58, 9.70, 9.74 [17:19:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.98, 9.96, 9.51 [17:19:41] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.12, 8.20, 7.86 [17:21:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.10, 9.24, 9.58 [17:21:35] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.11, 9.43, 8.34 [17:22:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.66, 9.10, 9.81 [17:23:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.42, 9.96, 9.65 [17:25:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 4.49, 7.92, 8.06 [17:25:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 15.46, 11.88, 10.37 [17:26:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.97, 1.32, 1.70 [17:27:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.83, 8.85, 9.23 [17:28:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.62, 3.06, 2.91 [17:28:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.78, 9.32, 9.63 [17:29:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.66, 3.37 [17:30:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.31, 3.16, 2.94 [17:31:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.33, 9.77, 9.66 [17:31:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 5.16, 9.15, 9.88 [17:31:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.60, 3.25, 3.25 [17:32:24] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.75, 3.10, 2.93 [17:33:10] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.02, 11.31, 8.94 [17:36:19] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.89, 3.00, 2.93 [17:37:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.17, 9.29, 8.67 [17:37:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.69, 6.94, 8.39 [17:40:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.82, 8.18, 7.46 [17:40:51] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.32, 9.30, 8.74 [17:42:31] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.39, 8.36, 7.62 [17:42:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.64, 9.36, 8.85 [17:43:05] !log [rhinos@mwtask111] finished deploy of {'world': True, 'l10n': True, 'ignoretime': True} to all - SUCCESS in 3455s [17:45:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.02, 7.37, 8.38 [17:46:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.62, 7.70, 8.30 [17:46:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.31, 8.93, 9.92 [17:48:10] !log [@test101] starting deploy of {'config': True} to all [17:48:11] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [17:51:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.29, 8.87, 8.44 [17:51:26] [02mw-config] 07RhinosF1 closed pull request 03#4543: Revert "temporarily disable babel until T9011 is figured out" - 13https://github.com/miraheze/mw-config/pull/4543 [17:51:28] [02miraheze/mw-config] 07RhinosF1 deleted branch 03revert-4542-Reception123-patch-1 [17:51:29] [02mw-config] 07RhinosF1 deleted branch 03revert-4542-Reception123-patch-1 - 13https://github.com/miraheze/mw-config [17:51:31] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8f2b351fe41e...a1513b35372b [17:51:32] [02miraheze/mw-config] 07RhinosF1 03a1513b3 - Revert "temporarily disable babel until T9011 is figured out (#4542)" (#4543) [17:52:32] miraheze/mw-config - RhinosF1 the build passed. [17:53:04] !log [@mwtask111] starting deploy of {'config': True} to all [17:53:14] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 10s [17:53:36] !log [@test101] starting deploy of {'config': True} to all [17:53:38] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [17:54:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.30, 9.08, 9.55 [17:57:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.93, 8.29, 8.36 [17:58:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.34, 8.48, 8.22 [18:00:06] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.81, 9.05, 8.44 [18:00:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.43, 9.58, 9.65 [18:01:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:02:28] .op [18:02:28] Attempting to OP... [18:02:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.18, 9.85, 9.74 [18:03:45] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.86, 3.99, 3.20 [18:03:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.92, 9.42, 8.77 [18:04:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.98, 3.72, 3.26 [18:04:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.78, 3.98, 3.39 [18:06:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.28, 4.05, 3.44 [18:06:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.36, 8.52, 8.46 [18:06:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.15, 4.43, 3.63 [18:06:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.95, 9.26, 9.59 [18:07:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.77, 3.86, 3.31 [18:08:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.70, 3.90, 3.45 [18:08:13] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.93, 8.17, 8.33 [18:09:38] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.10, 4.20, 3.50 [18:10:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 4.17, 3.60 [18:11:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:11:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.15, 3.74, 3.41 [18:12:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.80, 3.95, 3.59 [18:12:49] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.10, 1.52, 1.12 [18:13:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.15, 9.73, 9.18 [18:13:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.84, 3.58 [18:14:21] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.22, 8.61, 8.23 [18:14:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.98, 3.93, 3.74 [18:14:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.47, 9.08, 9.26 [18:14:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.88, 1.59, 1.19 [18:15:32] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.21, 3.04, 3.21 [18:15:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.77, 3.36, 3.43 [18:15:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.40, 9.03, 8.98 [18:16:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.63, 8.46, 8.23 [18:16:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.66, 1.60, 1.24 [18:17:53] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.70, 3.08, 3.31 [18:18:34] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.02, 2.96, 3.39 [18:18:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.97, 9.09, 9.26 [18:19:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.92, 9.64, 8.66 [18:21:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.63, 3.56, 3.45 [18:21:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.37, 9.10, 8.56 [18:22:48] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.11, 1.72, 1.37 [18:23:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.13, 3.09, 3.29 [18:24:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.76, 7.22, 8.39 [18:24:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.44, 1.62, 1.38 [18:26:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.44, 8.51, 8.34 [18:29:50] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.02, 8.17, 8.35 [18:29:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.19, 7.06, 8.11 [18:31:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:32:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.91, 3.40, 3.18 [18:34:06] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.07, 9.19, 8.71 [18:34:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:35:10] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/commit/4dc5289ccc10 [18:35:11] [02miraheze/ManageWiki] 07Universal-Omega 034dc5289 - Use texts type for `$wgNamespaceAliases` [18:35:13] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ManageWiki [18:35:15] [02ManageWiki] 07Universal-Omega opened pull request 03#347: Use texts type for `$wgNamespaceAliases` - 13https://github.com/miraheze/ManageWiki/pull/347 [18:38:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 5.91, 8.86, 8.79 [18:38:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.63, 2.84, 3.03 [18:38:42] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/4dc5289ccc10...735d10bb4589 [18:38:43] [02miraheze/ManageWiki] 07Universal-Omega 03735d10b - Update ManageWikiFormFactoryBuilder.php [18:38:45] [02ManageWiki] 07Universal-Omega synchronize pull request 03#347: Use texts type for `$wgNamespaceAliases` - 13https://github.com/miraheze/ManageWiki/pull/347 [18:40:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.50, 7.66, 8.35 [18:40:34] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 7.95, 12.77, 7.08 [18:40:42] PROBLEM - db121 MariaDB on db121 is CRITICAL: Can't connect to MySQL server on 'db121.miraheze.org' (115) [18:40:53] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1039 bytes in 0.282 second response time [18:42:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 3.64, 3.28 [18:43:26] miraheze/ManageWiki - Universal-Omega the build passed. [18:44:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.10, 3.29, 3.19 [18:44:26] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 0.18, 5.94, 5.54 [18:44:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:48:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.25, 3.56, 3.33 [18:52:23] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.90, 3.37, 3.31 [18:53:04] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [18:54:42] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 62 Threads: 32 Questions: 4238 Slow queries: 256 Opens: 857 Open tables: 851 Queries per second avg: 68.354 [18:54:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.75, 1.74, 1.47 [18:55:02] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 52003 bytes in 9.650 second response time [18:55:08] !log [void@db121] restart mariadb (oom-kill) [18:56:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.38, 1.62, 1.46 [18:57:07] Thanks void [18:57:09] And fast [18:57:25] If you can pop on irc, I wouldn't mind a quick chat with you [19:00:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:01:10] [discord] I'm not available for long, but sure [19:02:01] * Voidwalker waves [19:02:01] [tell] Voidwalker: 2022-03-24 - 04:11:43UTC tell Voidwalker night :) [19:02:02] [tell] Voidwalker: 2022-03-06 - 20:36:55UTC tell Voidwalker oh heh [19:03:07] Voidwalker: pm? [19:03:15] go ahead [19:05:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.62, 8.54, 7.42 [19:07:43] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.64, 8.11, 7.39 [19:10:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:19:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:22:38] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [19:22:38] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:23:41] [02CreateWiki] 07Universal-Omega synchronize pull request 03#312: Create new CreateWikiRegexConstraint class - 13https://github.com/miraheze/CreateWiki/pull/312 [19:23:42] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/ba6dcf809eed...13637b6abd73 [19:23:44] [02miraheze/CreateWiki] 07Universal-Omega 0313637b6 - Use single quotes [19:24:09] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/13637b6abd73...51f601bc5f9f [19:24:10] [02miraheze/CreateWiki] 07Universal-Omega 0351f601b - Add start and end to regexesFromMessage for validating [19:24:12] [02CreateWiki] 07Universal-Omega synchronize pull request 03#312: Create new CreateWikiRegexConstraint class - 13https://github.com/miraheze/CreateWiki/pull/312 [19:24:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:25:01] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/51f601bc5f9f...3d5324d8091e [19:25:02] [02miraheze/CreateWiki] 07Universal-Omega 033d5324d - Update CreateWikiRegexConstraint.php [19:25:04] [02CreateWiki] 07Universal-Omega synchronize pull request 03#312: Create new CreateWikiRegexConstraint class - 13https://github.com/miraheze/CreateWiki/pull/312 [19:29:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.67, 3.17, 2.82 [19:31:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.81, 3.00, 2.80 [19:35:50] miraheze/CreateWiki - Universal-Omega the build passed. [19:35:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.77, 8.15, 7.69 [19:37:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:39:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.37, 3.56, 3.11 [19:39:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.78, 7.80, 7.69 [19:41:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.05, 2.91, 2.92 [19:43:36] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.59, 9.42, 8.38 [19:44:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.56, 8.76, 7.80 [19:44:55] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.00, 3.84, 3.29 [19:45:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.91, 8.60, 8.21 [19:46:07] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.81, 3.32 [19:46:15] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.68, 8.33, 7.75 [19:46:53] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.31, 3.34, 3.18 [19:49:22] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.05, 8.16, 8.13 [19:50:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.64, 3.34 [19:50:07] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.26, 9.11, 8.19 [19:52:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.97, 8.67, 8.14 [19:52:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.69, 3.34, 3.27 [19:52:12] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [19:52:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:55:28] PROBLEM - test101 Puppet on test101 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [19:55:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.20, 8.24, 8.10 [19:56:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:02:23] PROBLEM - robloxapi.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'robloxapi.wiki' expires in 15 day(s) (Sun 17 Apr 2022 19:44:52 GMT +0000). [20:03:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.42, 8.65, 8.15 [20:05:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.20, 7.95, 7.94 [20:06:53] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cdb6992fa80b...d2c7414e7b4a [20:06:55] [02miraheze/ssl] 07MirahezeSSLBot 03d2c7414 - Bot: Update SSL cert for robloxapi.wiki [20:08:23] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.89, 1.44, 1.23 [20:08:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.56, 3.50, 3.17 [20:09:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.65, 9.14, 8.59 [20:10:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.98, 1.23, 1.17 [20:10:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.76, 3.23, 3.11 [20:11:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.44, 9.43, 8.79 [20:12:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.99, 4.92, 3.68 [20:13:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.90, 8.38, 8.47 [20:14:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.72, 3.38 [20:18:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.27, 3.25, 3.28 [20:21:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:21:45] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [20:21:46] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:22:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.15, 3.99, 3.54 [20:22:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.22, 3.55, 3.69 [20:23:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.99, 3.49, 3.12 [20:23:57] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.51, 1.63, 1.25 [20:24:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.57, 3.92, 3.80 [20:25:54] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.80, 1.25, 1.16 [20:26:00] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.19, 3.86, 2.04 [20:26:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:27:04] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.29, 3.61, 3.23 [20:29:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.50, 3.72, 3.33 [20:30:00] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.52, 1.95, 1.66 [20:31:17] RECOVERY - robloxapi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'robloxapi.wiki' will expire on Thu 30 Jun 2022 19:06:45 GMT +0000. [20:31:43] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.07, 1.95, 1.48 [20:32:00] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.97, 1.65, 1.58 [20:32:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.81, 3.36, 3.28 [20:33:40] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.12, 1.66, 1.42 [20:38:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 4.00, 4.00 [20:38:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.08, 3.72, 3.44 [20:39:29] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.40, 2.23, 1.72 [20:40:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.44, 4.37, 4.13 [20:40:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.99, 3.50, 3.40 [20:41:26] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.98, 1.74, 1.60 [20:41:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:42:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.35, 3.69, 3.46 [20:43:23] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.75, 1.36, 1.47 [20:44:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.56, 3.78, 3.53 [20:46:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.38, 3.25, 3.37 [20:47:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:48:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.07, 3.56, 3.96 [20:51:19] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [21:00:39] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.13, 3.98, 3.64 [21:00:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.65, 3.97 [21:00:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.28, 3.49, 3.67 [21:02:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:02:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.40, 4.32, 4.17 [21:04:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.32, 3.25, 3.56 [21:08:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.13, 3.80, 3.76 [21:08:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.26, 2.92, 3.37 [21:12:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.52, 3.72, 3.99 [21:13:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.80, 1.85, 1.48 [21:14:24] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.14, 3.43, 3.54 [21:15:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.59, 1.38, 1.35 [21:16:22] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.54, 3.04, 3.38 [21:18:41] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.07, 3.98, 3.97 [21:21:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.84, 3.54, 3.37 [21:22:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:23:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.09, 3.89, 3.50 [21:25:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.32, 3.56, 3.42 [21:27:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.85, 3.54 [21:29:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.27, 3.53, 3.45 [21:31:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.44, 3.40 [21:31:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.66, 3.82, 3.56 [21:32:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.52, 8.27, 7.64 [21:32:58] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.49, 3.79, 3.53 [21:34:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.13, 9.27, 8.20 [21:34:56] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.98, 3.81, 3.56 [21:36:04] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.94, 7.88, 7.64 [21:36:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.09, 8.64, 8.09 [21:38:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.85, 3.23, 3.37 [21:39:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.45, 3.70, 3.67 [21:39:56] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.17, 9.88, 8.52 [21:40:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.32, 8.50, 8.19 [21:41:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.86, 9.47, 8.54 [21:45:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.57, 4.38, 3.89 [21:45:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.32, 8.07, 8.18 [21:48:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.09, 3.69, 3.72 [21:50:26] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [21:50:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.42, 3.12, 3.91 [21:52:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:54:49] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.53, 2.80, 3.30 [22:00:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.58, 2.90, 3.37 [22:01:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:06:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:06:48] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.81, 1.66, 0.99 [22:14:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:29:20] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 3.98, 3.32 [22:30:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.99, 3.57, 3.18 [22:31:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.04, 3.16, 3.09 [22:32:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.33, 3.17, 3.08 [22:39:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:39:55] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.28, 4.10, 3.42 [22:41:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.71, 7.82, 7.30 [22:41:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.82, 3.82, 3.39 [22:43:21] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.85, 7.91, 7.40 [22:45:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.81, 3.47 [22:48:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.32, 3.05 [22:49:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.93, 3.47 [22:49:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:50:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.10, 2.87, 2.92 [22:51:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.35, 3.90, 3.49 [22:51:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.97, 7.78, 6.79 [22:53:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.61, 3.37, 3.35 [22:53:34] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.62, 7.21, 6.70 [22:53:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.43, 3.97, 3.78 [22:54:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:57:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.76, 2.16, 1.62 [22:59:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.86, 1.83, 1.55 [23:00:01] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:04] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:03:25] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.48, 3.05, 3.40 [23:05:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:06:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.51, 9.25, 8.10 [23:07:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.08, 1.59, 1.59 [23:08:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.10, 8.30, 7.89 [23:10:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:12:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.93, 9.23, 8.42 [23:16:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.88, 8.68, 7.82 [23:17:12] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1028s [23:18:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.19, 8.43, 8.36 [23:19:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.08, 8.95, 8.10 [23:21:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.53, 8.01, 7.85 [23:23:46] !log [@test101] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1424s [23:23:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:30:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.70, 3.84, 3.31 [23:32:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.57, 3.41, 3.22 [23:34:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.70, 3.03, 3.09 [23:35:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:40:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.78, 8.80, 8.00 [23:42:10] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.02, 8.04, 7.82 [23:42:56] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.11, 2.85 [23:42:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.04, 8.81, 8.13 [23:43:56] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.43, 8.69, 8.17 [23:44:53] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.68, 8.15, 7.97 [23:45:56] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.11, 8.39, 8.12 [23:46:55] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.51, 2.84, 2.80 [23:49:18] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.43, 1.80, 1.21 [23:51:18] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.66, 1.36, 1.12