[00:02:51] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.06, 1.74, 1.59 [00:06:45] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.37, 1.59, 1.56 [00:18:47] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:18:57] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:18:59] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:19:18] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:19:38] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:45] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.979 second response time [00:20:58] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.399 second response time [00:21:00] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.018 second response time [00:21:14] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.384 second response time [00:22:38] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:40] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:43] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.550 second response time [00:23:57] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:09] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:24] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.98, 10.52, 9.68 [00:30:05] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.498 second response time [00:30:16] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 2.551 second response time [00:30:21] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.691 second response time [00:30:55] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.507 second response time [00:30:57] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.320 second response time [00:32:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.89, 10.00, 9.65 [00:33:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.61, 1.66, 1.95 [00:35:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [00:36:36] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:36:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.92, 10.30, 9.88 [00:38:38] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 6.898 second response time [00:40:33] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.31, 11.31, 10.37 [00:42:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.73, 11.13, 10.42 [00:43:31] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.08, 1.33, 1.64 [00:48:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.23, 11.22, 10.62 [00:49:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.43, 2.06, 1.85 [00:50:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.59, 11.45, 10.79 [00:52:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/SpriteSheet-278f96a [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/334ee03497f5 [00:52:41] [02miraheze/mediawiki] 07dependabot[bot] 03334ee03 - Bump extensions/SpriteSheet from `55c5f2a` to `278f96a` [00:52:43] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/SpriteSheet-278f96a - 13https://github.com/miraheze/mediawiki [00:52:43] [url] Page not found · GitHub · GitHub | github.com [00:52:44] [02mediawiki] 07dependabot[bot] opened pull request 03#8160: Bump extensions/SpriteSheet from `55c5f2a` to `278f96a` - 13https://github.com/miraheze/mediawiki/pull/8160 [00:52:44] [url] Page not found · GitHub · GitHub | github.com [00:52:47] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MirahezeMagic-f7eb58a [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a9e5c6163c54 [00:52:48] [02miraheze/mediawiki] 07dependabot[bot] 03a9e5c61 - Bump extensions/MirahezeMagic from `d992b71` to `f7eb58a` [00:52:50] [02mediawiki] 07dependabot[bot] opened pull request 03#8161: Bump extensions/MirahezeMagic from `d992b71` to `f7eb58a` - 13https://github.com/miraheze/mediawiki/pull/8161 [00:52:51] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MirahezeMagic-f7eb58a - 13https://github.com/miraheze/mediawiki [00:52:53] [02mediawiki] 07dependabot[bot] commented on pull request 03#8082: Bump extensions/MirahezeMagic from `d992b71` to `dc01234` - 13https://github.com/miraheze/mediawiki/pull/8082#issuecomment-1186650660 [00:52:53] [url] Bump extensions/MirahezeMagic from `d992b71` to `dc01234` by dependabot[bot] · Pull Request #8082 · miraheze/mediawiki · GitHub | github.com [00:52:54] [02mediawiki] 07dependabot[bot] closed pull request 03#8082: Bump extensions/MirahezeMagic from `d992b71` to `dc01234` - 13https://github.com/miraheze/mediawiki/pull/8082 [00:52:55] [url] Page not found · GitHub · GitHub | github.com [00:52:56] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/MirahezeMagic-dc01234 [00:52:57] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/MirahezeMagic-dc01234 - 13https://github.com/miraheze/mediawiki [00:52:57] [url] Page not found · GitHub · GitHub | github.com [00:52:59] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/PDFEmbed-cd892de [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/3769f6363ce0 [00:53:00] [02miraheze/mediawiki] 07dependabot[bot] 033769f63 - Bump extensions/PDFEmbed from `d31da31` to `cd892de` [00:53:02] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/PDFEmbed-cd892de - 13https://github.com/miraheze/mediawiki [00:53:02] ... [00:53:03] [02mediawiki] 07dependabot[bot] opened pull request 03#8162: Bump extensions/PDFEmbed from `d31da31` to `cd892de` - 13https://github.com/miraheze/mediawiki/pull/8162 [00:53:04] [url] Page not found · GitHub · GitHub | github.com [00:53:05] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/CreateWiki-81917ce [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7b73295c8851 [00:53:06] [02miraheze/mediawiki] 07dependabot[bot] 037b73295 - Bump extensions/CreateWiki from `e96869b` to `81917ce` [00:53:08] [02mediawiki] 07dependabot[bot] opened pull request 03#8163: Bump extensions/CreateWiki from `e96869b` to `81917ce` - 13https://github.com/miraheze/mediawiki/pull/8163 [00:53:08] ... [00:53:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/CreateWiki-81917ce - 13https://github.com/miraheze/mediawiki [00:53:09] ... [00:53:11] [02mediawiki] 07dependabot[bot] commented on pull request 03#8094: Bump extensions/CreateWiki from `e96869b` to `33899c0` - 13https://github.com/miraheze/mediawiki/pull/8094#issuecomment-1186650723 [00:53:11] [url] Bump extensions/CreateWiki from `e96869b` to `33899c0` by dependabot[bot] · Pull Request #8094 · miraheze/mediawiki · GitHub | github.com [00:53:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/GlobalNewFiles-8a793af [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/3e9743d816bd [00:53:14] [02miraheze/mediawiki] 07dependabot[bot] 033e9743d - Bump extensions/GlobalNewFiles from `fa26fe1` to `8a793af` [00:53:15] [02mediawiki] 07dependabot[bot] closed pull request 03#8094: Bump extensions/CreateWiki from `e96869b` to `33899c0` - 13https://github.com/miraheze/mediawiki/pull/8094 [00:53:16] [url] Page not found · GitHub · GitHub | github.com [00:53:17] [02mediawiki] 07dependabot[bot] opened pull request 03#8164: Bump extensions/GlobalNewFiles from `fa26fe1` to `8a793af` - 13https://github.com/miraheze/mediawiki/pull/8164 [00:53:17] [url] Page not found · GitHub · GitHub | github.com [00:53:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/GlobalNewFiles-8a793af - 13https://github.com/miraheze/mediawiki [00:53:18] [url] Page not found · GitHub · GitHub | github.com [00:53:20] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/CreateWiki-33899c0 [00:53:21] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/CreateWiki-33899c0 - 13https://github.com/miraheze/mediawiki [00:53:21] [url] Page not found · GitHub · GitHub | github.com [00:53:23] [02mediawiki] 07dependabot[bot] commented on pull request 03#8087: Bump extensions/GlobalNewFiles from `fa26fe1` to `53a897b` - 13https://github.com/miraheze/mediawiki/pull/8087#issuecomment-1186650738 [00:53:23] [url] Bump extensions/GlobalNewFiles from `fa26fe1` to `53a897b` by dependabot[bot] · Pull Request #8087 · miraheze/mediawiki · GitHub | github.com [00:53:24] [02mediawiki] 07dependabot[bot] closed pull request 03#8087: Bump extensions/GlobalNewFiles from `fa26fe1` to `53a897b` - 13https://github.com/miraheze/mediawiki/pull/8087 [00:53:25] [url] Page not found · GitHub · GitHub | github.com [00:53:26] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/CheckUser-2bfb492 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/251d9beeb48d [00:53:27] [02miraheze/mediawiki] 07dependabot[bot] 03251d9be - Bump extensions/CheckUser from `bf74001` to `2bfb492` [00:53:29] [02mediawiki] 07dependabot[bot] opened pull request 03#8165: Bump extensions/CheckUser from `bf74001` to `2bfb492` - 13https://github.com/miraheze/mediawiki/pull/8165 [00:53:29] ... [00:53:30] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/GlobalNewFiles-53a897b [00:53:32] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/GlobalNewFiles-53a897b - 13https://github.com/miraheze/mediawiki [00:53:32] ... [00:53:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/CheckUser-2bfb492 - 13https://github.com/miraheze/mediawiki [00:53:33] ... [00:53:35] [02mediawiki] 07dependabot[bot] commented on pull request 03#8129: Bump extensions/CheckUser from `bf74001` to `fe9b1a2` - 13https://github.com/miraheze/mediawiki/pull/8129#issuecomment-1186650774 [00:53:35] [url] Bump extensions/CheckUser from `bf74001` to `fe9b1a2` by dependabot[bot] · Pull Request #8129 · miraheze/mediawiki · GitHub | github.com [00:53:36] [02mediawiki] 07dependabot[bot] closed pull request 03#8129: Bump extensions/CheckUser from `bf74001` to `fe9b1a2` - 13https://github.com/miraheze/mediawiki/pull/8129 [00:53:37] [url] Page not found · GitHub · GitHub | github.com [00:53:38] [02mediawiki] 07github-actions[bot] labeled pull request 03#8162: Bump extensions/PDFEmbed from `d31da31` to `cd892de` - 13https://github.com/miraheze/mediawiki/pull/8162 [00:53:38] [url] Page not found · GitHub · GitHub | github.com [00:53:39] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/CheckUser-fe9b1a2 [00:53:41] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/CheckUser-fe9b1a2 - 13https://github.com/miraheze/mediawiki [00:53:41] [url] Page not found · GitHub · GitHub | github.com [00:53:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#8161: Bump extensions/MirahezeMagic from `d992b71` to `f7eb58a` - 13https://github.com/miraheze/mediawiki/pull/8161 [00:53:43] [url] Page not found · GitHub · GitHub | github.com [00:53:44] [02mediawiki] 07github-actions[bot] labeled pull request 03#8161: Bump extensions/MirahezeMagic from `d992b71` to `f7eb58a` - 13https://github.com/miraheze/mediawiki/pull/8161 [00:53:44] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:53:44] [url] Page not found · GitHub · GitHub | github.com [00:53:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#8160: Bump extensions/SpriteSheet from `55c5f2a` to `278f96a` - 13https://github.com/miraheze/mediawiki/pull/8160 [00:53:46] ... [00:53:47] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ManageWiki-aad1060 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/47e6cd538116 [00:53:48] [02miraheze/mediawiki] 07dependabot[bot] 0347e6cd5 - Bump extensions/ManageWiki from `b0139c2` to `aad1060` [00:53:50] [02mediawiki] 07dependabot[bot] opened pull request 03#8166: Bump extensions/ManageWiki from `b0139c2` to `aad1060` - 13https://github.com/miraheze/mediawiki/pull/8166 [00:53:50] ... [00:53:51] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ManageWiki-aad1060 - 13https://github.com/miraheze/mediawiki [00:53:51] ... [00:53:53] [02mediawiki] 07dependabot[bot] commented on pull request 03#8104: Bump extensions/ManageWiki from `b0139c2` to `202eaec` - 13https://github.com/miraheze/mediawiki/pull/8104#issuecomment-1186650829 [00:53:53] [url] Bump extensions/ManageWiki from `b0139c2` to `202eaec` by dependabot[bot] · Pull Request #8104 · miraheze/mediawiki · GitHub | github.com [00:53:54] [02mediawiki] 07dependabot[bot] closed pull request 03#8104: Bump extensions/ManageWiki from `b0139c2` to `202eaec` - 13https://github.com/miraheze/mediawiki/pull/8104 [00:53:55] [url] Page not found · GitHub · GitHub | github.com [00:53:56] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/ManageWiki-202eaec [00:53:57] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/ManageWiki-202eaec - 13https://github.com/miraheze/mediawiki [00:53:57] [url] Page not found · GitHub · GitHub | github.com [00:53:59] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/WikiDiscover-6cd4a2b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/fa8a74e54bbb [00:54:00] [02miraheze/mediawiki] 07dependabot[bot] 03fa8a74e - Bump extensions/WikiDiscover from `0bd18ba` to `6cd4a2b` [00:54:02] [02mediawiki] 07dependabot[bot] opened pull request 03#8167: Bump extensions/WikiDiscover from `0bd18ba` to `6cd4a2b` - 13https://github.com/miraheze/mediawiki/pull/8167 [00:54:02] [url] Page not found · GitHub · GitHub | github.com [00:54:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/WikiDiscover-6cd4a2b - 13https://github.com/miraheze/mediawiki [00:54:03] [url] Page not found · GitHub · GitHub | github.com [00:54:05] [02mediawiki] 07github-actions[bot] labeled pull request 03#8163: Bump extensions/CreateWiki from `e96869b` to `81917ce` - 13https://github.com/miraheze/mediawiki/pull/8163 [00:54:05] [url] Page not found · GitHub · GitHub | github.com [00:54:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#8163: Bump extensions/CreateWiki from `e96869b` to `81917ce` - 13https://github.com/miraheze/mediawiki/pull/8163 [00:54:06] ... [00:54:08] [02mediawiki] 07dependabot[bot] commented on pull request 03#8098: Bump extensions/WikiDiscover from `0bd18ba` to `76c8b43` - 13https://github.com/miraheze/mediawiki/pull/8098#issuecomment-1186650879 [00:54:08] [url] Bump extensions/WikiDiscover from `0bd18ba` to `76c8b43` by dependabot[bot] · Pull Request #8098 · miraheze/mediawiki · GitHub | github.com [00:54:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/PageForms-1888d91 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/95ebcb0ced0a [00:54:11] [02miraheze/mediawiki] 07dependabot[bot] 0395ebcb0 - Bump extensions/PageForms from `7730130` to `1888d91` [00:54:12] [02mediawiki] 07dependabot[bot] closed pull request 03#8098: Bump extensions/WikiDiscover from `0bd18ba` to `76c8b43` - 13https://github.com/miraheze/mediawiki/pull/8098 [00:54:13] ... [00:54:14] [02mediawiki] 07dependabot[bot] opened pull request 03#8168: Bump extensions/PageForms from `7730130` to `1888d91` - 13https://github.com/miraheze/mediawiki/pull/8168 [00:54:14] ... [00:54:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/PageForms-1888d91 - 13https://github.com/miraheze/mediawiki [00:54:15] [url] Page not found · GitHub · GitHub | github.com [00:54:17] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/WikiDiscover-76c8b43 - 13https://github.com/miraheze/mediawiki [00:54:17] [url] Page not found · GitHub · GitHub | github.com [00:54:18] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/WikiDiscover-76c8b43 [00:54:20] [02mediawiki] 07dependabot[bot] closed pull request 03#8133: Bump extensions/PageForms from `7730130` to `9dc2162` - 13https://github.com/miraheze/mediawiki/pull/8133 [00:54:20] [url] Page not found · GitHub · GitHub | github.com [00:54:21] [02mediawiki] 07github-actions[bot] labeled pull request 03#8164: Bump extensions/GlobalNewFiles from `fa26fe1` to `8a793af` - 13https://github.com/miraheze/mediawiki/pull/8164 [00:54:22] [url] Page not found · GitHub · GitHub | github.com [00:54:23] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/PageForms-9dc2162 - 13https://github.com/miraheze/mediawiki [00:54:23] [url] Page not found · GitHub · GitHub | github.com [00:54:24] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/PageForms-9dc2162 [00:54:26] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Citizen-1bf51fc [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ebce8b906bd5 [00:54:27] [02miraheze/mediawiki] 07dependabot[bot] 03ebce8b9 - Bump skins/Citizen from `e53afa5` to `1bf51fc` [00:54:29] [02mediawiki] 07dependabot[bot] commented on pull request 03#8133: Bump extensions/PageForms from `7730130` to `9dc2162` - 13https://github.com/miraheze/mediawiki/pull/8133#issuecomment-1186650904 [00:54:29] [url] Bump extensions/PageForms from `7730130` to `9dc2162` by dependabot[bot] · Pull Request #8133 · miraheze/mediawiki · GitHub | github.com [00:54:30] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Citizen-1bf51fc - 13https://github.com/miraheze/mediawiki [00:54:30] ... [00:54:32] [02mediawiki] 07dependabot[bot] commented on pull request 03#8156: Bump skins/Citizen from `e53afa5` to `7b2a046` - 13https://github.com/miraheze/mediawiki/pull/8156#issuecomment-1186650937 [00:54:32] [url] Bump skins/Citizen from `e53afa5` to `7b2a046` by dependabot[bot] · Pull Request #8156 · miraheze/mediawiki · GitHub | github.com [00:54:33] [02mediawiki] 07dependabot[bot] opened pull request 03#8169: Bump skins/Citizen from `e53afa5` to `1bf51fc` - 13https://github.com/miraheze/mediawiki/pull/8169 [00:54:34] ... [00:54:35] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Citizen-7b2a046 [00:54:36] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Citizen-7b2a046 - 13https://github.com/miraheze/mediawiki [00:54:36] [url] Page not found · GitHub · GitHub | github.com [00:54:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/SocialProfile-b2e71bb [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/06edbc53535d [00:54:39] [02miraheze/mediawiki] 07dependabot[bot] 0306edbc5 - Bump extensions/SocialProfile from `f5cab34` to `b2e71bb` [00:54:41] [02mediawiki] 07dependabot[bot] opened pull request 03#8170: Bump extensions/SocialProfile from `f5cab34` to `b2e71bb` - 13https://github.com/miraheze/mediawiki/pull/8170 [00:54:41] [url] Page not found · GitHub · GitHub | github.com [00:54:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/SocialProfile-b2e71bb - 13https://github.com/miraheze/mediawiki [00:54:42] [url] Page not found · GitHub · GitHub | github.com [00:54:44] [02mediawiki] 07dependabot[bot] closed pull request 03#8156: Bump skins/Citizen from `e53afa5` to `7b2a046` - 13https://github.com/miraheze/mediawiki/pull/8156 [00:54:44] [url] Page not found · GitHub · GitHub | github.com [00:54:45] [02mediawiki] 07dependabot[bot] commented on pull request 03#8136: Bump extensions/SocialProfile from `f5cab34` to `acfd842` - 13https://github.com/miraheze/mediawiki/pull/8136#issuecomment-1186650958 [00:54:46] [url] Bump extensions/SocialProfile from `f5cab34` to `acfd842` by dependabot[bot] · Pull Request #8136 · miraheze/mediawiki · GitHub | github.com [00:54:47] [02mediawiki] 07dependabot[bot] closed pull request 03#8136: Bump extensions/SocialProfile from `f5cab34` to `acfd842` - 13https://github.com/miraheze/mediawiki/pull/8136 [00:54:47] [url] Page not found · GitHub · GitHub | github.com [00:54:48] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/SocialProfile-acfd842 [00:54:50] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/SocialProfile-acfd842 - 13https://github.com/miraheze/mediawiki [00:54:50] ... [00:54:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/RemovePII-656c257 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e9b6be5cde97 [00:54:53] [02miraheze/mediawiki] 07dependabot[bot] 03e9b6be5 - Bump extensions/RemovePII from `3be5c88` to `656c257` [00:54:54] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/RemovePII-656c257 - 13https://github.com/miraheze/mediawiki [00:54:54] ... [00:54:56] [02mediawiki] 07dependabot[bot] opened pull request 03#8171: Bump extensions/RemovePII from `3be5c88` to `656c257` - 13https://github.com/miraheze/mediawiki/pull/8171 [00:54:56] ... [00:54:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#8166: Bump extensions/ManageWiki from `b0139c2` to `aad1060` - 13https://github.com/miraheze/mediawiki/pull/8166 [00:54:58] [url] Page not found · GitHub · GitHub | github.com [00:54:59] [02mediawiki] 07dependabot[bot] commented on pull request 03#8121: Bump extensions/RemovePII from `3be5c88` to `538bafc` - 13https://github.com/miraheze/mediawiki/pull/8121#issuecomment-1186650996 [00:55:00] [url] Bump extensions/RemovePII from `3be5c88` to `538bafc` by dependabot[bot] · Pull Request #8121 · miraheze/mediawiki · GitHub | github.com [00:55:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Video-aadd04f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/6095cd0f2676 [00:55:02] [02miraheze/mediawiki] 07dependabot[bot] 036095cd0 - Bump extensions/Video from `f8467b5` to `aadd04f` [00:55:03] [02mediawiki] 07dependabot[bot] opened pull request 03#8172: Bump extensions/Video from `f8467b5` to `aadd04f` - 13https://github.com/miraheze/mediawiki/pull/8172 [00:55:04] [url] Page not found · GitHub · GitHub | github.com [00:55:05] [02mediawiki] 07dependabot[bot] closed pull request 03#8121: Bump extensions/RemovePII from `3be5c88` to `538bafc` - 13https://github.com/miraheze/mediawiki/pull/8121 [00:55:05] [url] Page not found · GitHub · GitHub | github.com [00:55:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Video-aadd04f - 13https://github.com/miraheze/mediawiki [00:55:06] [url] Page not found · GitHub · GitHub | github.com [00:55:08] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/RemovePII-538bafc [00:55:09] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/RemovePII-538bafc - 13https://github.com/miraheze/mediawiki [00:55:09] [url] Page not found · GitHub · GitHub | github.com [00:55:11] [02mediawiki] 07github-actions[bot] labeled pull request 03#8167: Bump extensions/WikiDiscover from `0bd18ba` to `6cd4a2b` - 13https://github.com/miraheze/mediawiki/pull/8167 [00:55:11] ... [00:55:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/UniversalLanguageSelector-e36beb0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1f7afba7da7b [00:55:14] [02miraheze/mediawiki] 07dependabot[bot] 031f7afba - Bump extensions/UniversalLanguageSelector from `b1b48bd` to `e36beb0` [00:55:15] [02mediawiki] 07dependabot[bot] opened pull request 03#8173: Bump extensions/UniversalLanguageSelector from `b1b48bd` to `e36beb0` - 13https://github.com/miraheze/mediawiki/pull/8173 [00:55:16] ... [00:55:17] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/UniversalLanguageSelector-e36beb0 - 13https://github.com/miraheze/mediawiki [00:55:17] ... [00:55:18] [02mediawiki] 07dependabot[bot] commented on pull request 03#8137: Bump extensions/UniversalLanguageSelector from `b1b48bd` to `26ea437` - 13https://github.com/miraheze/mediawiki/pull/8137#issuecomment-1186651073 [00:55:19] [url] Bump extensions/UniversalLanguageSelector from `b1b48bd` to `26ea437` by dependabot[bot] · Pull Request #8137 · miraheze/mediawiki · GitHub | github.com [00:55:20] [02mediawiki] 07dependabot[bot] closed pull request 03#8137: Bump extensions/UniversalLanguageSelector from `b1b48bd` to `26ea437` - 13https://github.com/miraheze/mediawiki/pull/8137 [00:55:20] [url] Page not found · GitHub · GitHub | github.com [00:55:21] [02mediawiki] 07github-actions[bot] labeled pull request 03#8168: Bump extensions/PageForms from `7730130` to `1888d91` - 13https://github.com/miraheze/mediawiki/pull/8168 [00:55:22] [url] Page not found · GitHub · GitHub | github.com [00:55:23] [02mediawiki] 07github-actions[bot] labeled pull request 03#8168: Bump extensions/PageForms from `7730130` to `1888d91` - 13https://github.com/miraheze/mediawiki/pull/8168 [00:55:23] [url] Page not found · GitHub · GitHub | github.com [00:55:24] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/UniversalLanguageSelector-26ea437 [00:55:26] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/UniversalLanguageSelector-26ea437 - 13https://github.com/miraheze/mediawiki [00:55:26] [url] Page not found · GitHub · GitHub | github.com [00:55:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/RottenLinks-d6de3ca [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/352100600038 [00:55:28] [url] Page not found · GitHub · GitHub | github.com [00:55:29] [02miraheze/mediawiki] 07dependabot[bot] 033521006 - Bump extensions/RottenLinks from `5552b0f` to `d6de3ca` [00:55:30] [02mediawiki] 07dependabot[bot] opened pull request 03#8174: Bump extensions/RottenLinks from `5552b0f` to `d6de3ca` - 13https://github.com/miraheze/mediawiki/pull/8174 [00:55:31] ... [00:55:32] [02mediawiki] 07github-actions[bot] labeled pull request 03#8168: Bump extensions/PageForms from `7730130` to `1888d91` - 13https://github.com/miraheze/mediawiki/pull/8168 [00:55:32] ... [00:55:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/RottenLinks-d6de3ca - 13https://github.com/miraheze/mediawiki [00:55:33] ... [00:55:35] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/DataDump-32a3651 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b10d67dda9ec [00:55:36] [02miraheze/mediawiki] 07dependabot[bot] 03b10d67d - Bump extensions/DataDump from `58eedfa` to `32a3651` [00:55:38] [02mediawiki] 07github-actions[bot] labeled pull request 03#8169: Bump skins/Citizen from `e53afa5` to `1bf51fc` - 13https://github.com/miraheze/mediawiki/pull/8169 [00:55:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#8170: Bump extensions/SocialProfile from `f5cab34` to `b2e71bb` - 13https://github.com/miraheze/mediawiki/pull/8170 [00:55:41] [02mediawiki] 07github-actions[bot] labeled pull request 03#8172: Bump extensions/Video from `f8467b5` to `aadd04f` - 13https://github.com/miraheze/mediawiki/pull/8172 [00:55:42] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.304 second response time [00:55:42] [02mediawiki] 07dependabot[bot] opened pull request 03#8175: Bump extensions/DataDump from `58eedfa` to `32a3651` - 13https://github.com/miraheze/mediawiki/pull/8175 [00:55:44] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/DataDump-32a3651 - 13https://github.com/miraheze/mediawiki [00:55:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#8169: Bump skins/Citizen from `e53afa5` to `1bf51fc` - 13https://github.com/miraheze/mediawiki/pull/8169 [00:55:47] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/DynamicPageList3-3b85f45 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5fe1a8fc2eb0 [00:55:48] [02miraheze/mediawiki] 07dependabot[bot] 035fe1a8f - Bump extensions/DynamicPageList3 from `bbd4923` to `3b85f45` [00:55:50] [02mediawiki] 07dependabot[bot] opened pull request 03#8176: Bump extensions/DynamicPageList3 from `bbd4923` to `3b85f45` - 13https://github.com/miraheze/mediawiki/pull/8176 [00:55:51] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/DynamicPageList3-3b85f45 - 13https://github.com/miraheze/mediawiki [00:55:53] [02mediawiki] 07github-actions[bot] labeled pull request 03#8171: Bump extensions/RemovePII from `3be5c88` to `656c257` - 13https://github.com/miraheze/mediawiki/pull/8171 [00:55:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/YouTube-8d1355e [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/af5f794c6f74 [00:55:56] [02miraheze/mediawiki] 07dependabot[bot] 03af5f794 - Bump extensions/YouTube from `b199c95` to `8d1355e` [00:55:57] [02mediawiki] 07dependabot[bot] opened pull request 03#8177: Bump extensions/YouTube from `b199c95` to `8d1355e` - 13https://github.com/miraheze/mediawiki/pull/8177 [00:55:59] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/YouTube-8d1355e - 13https://github.com/miraheze/mediawiki [00:56:00] [02mediawiki] 07dependabot[bot] commented on pull request 03#8128: Bump extensions/DynamicPageList3 from `bbd4923` to `e1c23d2` - 13https://github.com/miraheze/mediawiki/pull/8128#issuecomment-1186651189 [00:56:01] [url] Bump extensions/DynamicPageList3 from `bbd4923` to `e1c23d2` by dependabot[bot] · Pull Request #8128 · miraheze/mediawiki · GitHub | github.com [00:56:02] [02mediawiki] 07dependabot[bot] closed pull request 03#8128: Bump extensions/DynamicPageList3 from `bbd4923` to `e1c23d2` - 13https://github.com/miraheze/mediawiki/pull/8128 [00:56:02] [url] Page not found · GitHub · GitHub | github.com [00:56:03] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/DynamicPageList3-e1c23d2 [00:56:05] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/DynamicPageList3-e1c23d2 - 13https://github.com/miraheze/mediawiki [00:56:05] [url] Page not found · GitHub · GitHub | github.com [00:56:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#8174: Bump extensions/RottenLinks from `5552b0f` to `d6de3ca` - 13https://github.com/miraheze/mediawiki/pull/8174 [00:56:07] [url] Page not found · GitHub · GitHub | github.com [00:56:08] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Evelution-4c75c59 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4a02a45be045 [00:56:09] [02miraheze/mediawiki] 07dependabot[bot] 034a02a45 - Bump skins/Evelution from `631cf5e` to `4c75c59` [00:56:11] [02mediawiki] 07dependabot[bot] opened pull request 03#8178: Bump skins/Evelution from `631cf5e` to `4c75c59` - 13https://github.com/miraheze/mediawiki/pull/8178 [00:56:11] [url] Page not found · GitHub · GitHub | github.com [00:56:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Evelution-4c75c59 - 13https://github.com/miraheze/mediawiki [00:56:12] [url] Page not found · GitHub · GitHub | github.com [00:56:14] [02mediawiki] 07github-actions[bot] labeled pull request 03#8173: Bump extensions/UniversalLanguageSelector from `b1b48bd` to `e36beb0` - 13https://github.com/miraheze/mediawiki/pull/8173 [00:56:14] ... [00:56:15] [02mediawiki] 07dependabot[bot] commented on pull request 03#8148: Bump skins/Evelution from `631cf5e` to `f08f4e4` - 13https://github.com/miraheze/mediawiki/pull/8148#issuecomment-1186651256 [00:56:16] [url] Bump skins/Evelution from `631cf5e` to `f08f4e4` by dependabot[bot] · Pull Request #8148 · miraheze/mediawiki · GitHub | github.com [00:56:17] [02mediawiki] 07dependabot[bot] closed pull request 03#8148: Bump skins/Evelution from `631cf5e` to `f08f4e4` - 13https://github.com/miraheze/mediawiki/pull/8148 [00:56:17] ... [00:56:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Translate-3c1470f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e64ec6a41813 [00:56:20] [02miraheze/mediawiki] 07dependabot[bot] 03e64ec6a - Bump extensions/Translate from `d7618b5` to `3c1470f` [00:56:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.11, 11.66, 11.05 [00:56:21] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Evelution-f08f4e4 [00:56:23] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Translate-3c1470f - 13https://github.com/miraheze/mediawiki [00:56:23] ... [00:56:24] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Evelution-f08f4e4 - 13https://github.com/miraheze/mediawiki [00:56:24] [url] Page not found · GitHub · GitHub | github.com [00:56:26] [02mediawiki] 07dependabot[bot] opened pull request 03#8179: Bump extensions/Translate from `d7618b5` to `3c1470f` - 13https://github.com/miraheze/mediawiki/pull/8179 [00:56:26] [url] Page not found · GitHub · GitHub | github.com [00:56:27] [02mediawiki] 07dependabot[bot] commented on pull request 03#8146: Bump extensions/Translate from `d7618b5` to `97c2f74` - 13https://github.com/miraheze/mediawiki/pull/8146#issuecomment-1186651288 [00:56:28] [url] Bump extensions/Translate from `d7618b5` to `97c2f74` by dependabot[bot] · Pull Request #8146 · miraheze/mediawiki · GitHub | github.com [00:56:29] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Translate-97c2f74 [00:56:30] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Translate-97c2f74 - 13https://github.com/miraheze/mediawiki [00:56:30] [url] Page not found · GitHub · GitHub | github.com [00:56:32] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MatomoAnalytics-3d550db [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/bf974ee1f0c8 [00:56:33] [02miraheze/mediawiki] 07dependabot[bot] 03bf974ee - Bump extensions/MatomoAnalytics from `79ddfa3` to `3d550db` [00:56:35] [02mediawiki] 07dependabot[bot] closed pull request 03#8146: Bump extensions/Translate from `d7618b5` to `97c2f74` - 13https://github.com/miraheze/mediawiki/pull/8146 [00:56:35] [url] Page not found · GitHub · GitHub | github.com [00:56:36] [02mediawiki] 07dependabot[bot] opened pull request 03#8180: Bump extensions/MatomoAnalytics from `79ddfa3` to `3d550db` - 13https://github.com/miraheze/mediawiki/pull/8180 [00:56:37] [url] Page not found · GitHub · GitHub | github.com [00:56:38] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MatomoAnalytics-3d550db - 13https://github.com/miraheze/mediawiki [00:56:38] ... [00:56:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#8175: Bump extensions/DataDump from `58eedfa` to `32a3651` - 13https://github.com/miraheze/mediawiki/pull/8175 [00:56:40] ... [00:56:41] [02mediawiki] 07dependabot[bot] commented on pull request 03#8122: Bump extensions/MatomoAnalytics from `79ddfa3` to `5faa403` - 13https://github.com/miraheze/mediawiki/pull/8122#issuecomment-1186651325 [00:56:41] [url] Bump extensions/MatomoAnalytics from `79ddfa3` to `5faa403` by dependabot[bot] · Pull Request #8122 · miraheze/mediawiki · GitHub | github.com [00:56:42] [02mediawiki] 07dependabot[bot] closed pull request 03#8122: Bump extensions/MatomoAnalytics from `79ddfa3` to `5faa403` - 13https://github.com/miraheze/mediawiki/pull/8122 [00:56:43] [url] Page not found · GitHub · GitHub | github.com [00:56:44] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/MatomoAnalytics-5faa403 [00:56:45] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/MatomoAnalytics-5faa403 - 13https://github.com/miraheze/mediawiki [00:56:45] [url] Page not found · GitHub · GitHub | github.com [00:56:47] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ImportDump-62f7e59 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e47852b4f86d [00:56:48] [02miraheze/mediawiki] 07dependabot[bot] 03e47852b - Bump extensions/ImportDump from `1d4b14b` to `62f7e59` [00:56:50] [02mediawiki] 07github-actions[bot] labeled pull request 03#8176: Bump extensions/DynamicPageList3 from `bbd4923` to `3b85f45` - 13https://github.com/miraheze/mediawiki/pull/8176 [00:56:50] ... [00:56:51] [02mediawiki] 07dependabot[bot] opened pull request 03#8181: Bump extensions/ImportDump from `1d4b14b` to `62f7e59` - 13https://github.com/miraheze/mediawiki/pull/8181 [00:56:52] [url] Page not found · GitHub · GitHub | github.com [00:56:53] [02mediawiki] 07github-actions[bot] labeled pull request 03#8176: Bump extensions/DynamicPageList3 from `bbd4923` to `3b85f45` - 13https://github.com/miraheze/mediawiki/pull/8176 [00:56:53] [url] Page not found · GitHub · GitHub | github.com [00:56:54] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ImportDump-62f7e59 - 13https://github.com/miraheze/mediawiki [00:56:55] [url] Page not found · GitHub · GitHub | github.com [00:56:56] [02mediawiki] 07github-actions[bot] labeled pull request 03#8177: Bump extensions/YouTube from `b199c95` to `8d1355e` - 13https://github.com/miraheze/mediawiki/pull/8177 [00:56:56] ... [00:56:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#8165: Bump extensions/CheckUser from `bf74001` to `2bfb492` - 13https://github.com/miraheze/mediawiki/pull/8165 [00:56:58] ... [00:56:59] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/ImportDump-e80a736 [00:57:00] [02mediawiki] 07dependabot[bot] commented on pull request 03#8141: Bump extensions/ImportDump from `1d4b14b` to `e80a736` - 13https://github.com/miraheze/mediawiki/pull/8141#issuecomment-1186651383 [00:57:01] [url] Bump extensions/ImportDump from `1d4b14b` to `e80a736` by dependabot[bot] · Pull Request #8141 · miraheze/mediawiki · GitHub | github.com [00:57:02] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/ImportDump-e80a736 - 13https://github.com/miraheze/mediawiki [00:57:02] [url] Page not found · GitHub · GitHub | github.com [00:57:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#8165: Bump extensions/CheckUser from `bf74001` to `2bfb492` - 13https://github.com/miraheze/mediawiki/pull/8165 [00:57:03] [url] Page not found · GitHub · GitHub | github.com [00:57:05] [02mediawiki] 07dependabot[bot] closed pull request 03#8141: Bump extensions/ImportDump from `1d4b14b` to `e80a736` - 13https://github.com/miraheze/mediawiki/pull/8141 [00:57:05] ... [00:57:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#8180: Bump extensions/MatomoAnalytics from `79ddfa3` to `3d550db` - 13https://github.com/miraheze/mediawiki/pull/8180 [00:57:07] [url] Page not found · GitHub · GitHub | github.com [00:57:08] [02mediawiki] 07github-actions[bot] labeled pull request 03#8180: Bump extensions/MatomoAnalytics from `79ddfa3` to `3d550db` - 13https://github.com/miraheze/mediawiki/pull/8180 [00:57:08] [url] Page not found · GitHub · GitHub | github.com [00:57:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MintyDocs-3b8c94a [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8c1c9e1dd500 [00:57:11] [02miraheze/mediawiki] 07dependabot[bot] 038c1c9e1 - Bump extensions/MintyDocs from `b39484a` to `3b8c94a` [00:57:12] [02mediawiki] 07dependabot[bot] opened pull request 03#8182: Bump extensions/MintyDocs from `b39484a` to `3b8c94a` - 13https://github.com/miraheze/mediawiki/pull/8182 [00:57:13] [url] Page not found · GitHub · GitHub | github.com [00:57:14] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MintyDocs-3b8c94a - 13https://github.com/miraheze/mediawiki [00:57:14] ... [00:57:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#8178: Bump skins/Evelution from `631cf5e` to `4c75c59` - 13https://github.com/miraheze/mediawiki/pull/8178 [00:57:16] ... [00:57:17] [02mediawiki] 07dependabot[bot] commented on pull request 03#8125: Bump extensions/MintyDocs from `b39484a` to `38f5e0f` - 13https://github.com/miraheze/mediawiki/pull/8125#issuecomment-1186651426 [00:57:17] [url] Bump extensions/MintyDocs from `b39484a` to `38f5e0f` by dependabot[bot] · Pull Request #8125 · miraheze/mediawiki · GitHub | github.com [00:57:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/IncidentReporting-394dfbd [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f4bab4e8ba83 [00:57:20] [02miraheze/mediawiki] 07dependabot[bot] 03f4bab4e - Bump extensions/IncidentReporting from `a6680c0` to `394dfbd` [00:57:21] [02mediawiki] 07dependabot[bot] opened pull request 03#8183: Bump extensions/IncidentReporting from `a6680c0` to `394dfbd` - 13https://github.com/miraheze/mediawiki/pull/8183 [00:57:22] [url] Page not found · GitHub · GitHub | github.com [00:57:23] [02mediawiki] 07dependabot[bot] closed pull request 03#8125: Bump extensions/MintyDocs from `b39484a` to `38f5e0f` - 13https://github.com/miraheze/mediawiki/pull/8125 [00:57:23] [url] Page not found · GitHub · GitHub | github.com [00:57:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/IncidentReporting-394dfbd - 13https://github.com/miraheze/mediawiki [00:57:25] ... [00:57:26] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/MintyDocs-38f5e0f [00:57:27] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/MintyDocs-38f5e0f - 13https://github.com/miraheze/mediawiki [00:57:28] [url] Page not found · GitHub · GitHub | github.com [00:57:29] [02mediawiki] 07dependabot[bot] commented on pull request 03#8079: Bump extensions/IncidentReporting from `a6680c0` to `5eebe1a` - 13https://github.com/miraheze/mediawiki/pull/8079#issuecomment-1186651457 [00:57:29] [url] Bump extensions/IncidentReporting from `a6680c0` to `5eebe1a` by dependabot[bot] · Pull Request #8079 · miraheze/mediawiki · GitHub | github.com [00:57:30] [02mediawiki] 07dependabot[bot] closed pull request 03#8079: Bump extensions/IncidentReporting from `a6680c0` to `5eebe1a` - 13https://github.com/miraheze/mediawiki/pull/8079 [00:57:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.34, 1.81, 1.88 [00:57:31] [url] Page not found · GitHub · GitHub | github.com [00:57:32] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/IncidentReporting-5eebe1a [00:57:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#8179: Bump extensions/Translate from `d7618b5` to `3c1470f` - 13https://github.com/miraheze/mediawiki/pull/8179 [00:57:34] [url] Page not found · GitHub · GitHub | github.com [00:57:35] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/IncidentReporting-5eebe1a - 13https://github.com/miraheze/mediawiki [00:57:35] ... [00:57:36] [02mediawiki] 07github-actions[bot] labeled pull request 03#8179: Bump extensions/Translate from `d7618b5` to `3c1470f` - 13https://github.com/miraheze/mediawiki/pull/8179 [00:57:37] ... [00:57:38] [02mediawiki] 07github-actions[bot] labeled pull request 03#8179: Bump extensions/Translate from `d7618b5` to `3c1470f` - 13https://github.com/miraheze/mediawiki/pull/8179 [00:57:38] [url] Page not found · GitHub · GitHub | github.com [00:57:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#8181: Bump extensions/ImportDump from `1d4b14b` to `62f7e59` - 13https://github.com/miraheze/mediawiki/pull/8181 [00:57:40] [url] Page not found · GitHub · GitHub | github.com [00:57:41] [02mediawiki] 07github-actions[bot] labeled pull request 03#8181: Bump extensions/ImportDump from `1d4b14b` to `62f7e59` - 13https://github.com/miraheze/mediawiki/pull/8181 [00:57:41] ... [00:57:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#8182: Bump extensions/MintyDocs from `b39484a` to `3b8c94a` - 13https://github.com/miraheze/mediawiki/pull/8182 [00:57:43] [url] Page not found · GitHub · GitHub | github.com [00:57:44] [02mediawiki] 07github-actions[bot] labeled pull request 03#8183: Bump extensions/IncidentReporting from `a6680c0` to `394dfbd` - 13https://github.com/miraheze/mediawiki/pull/8183 [00:58:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.23, 11.00, 10.90 [01:01:09] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:01:34] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:01:38] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:02:02] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:02:03] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:03:10] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.971 second response time [01:03:35] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.457 second response time [01:03:42] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.581 second response time [01:04:02] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 4.672 second response time [01:04:05] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.662 second response time [01:06:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.68, 11.55, 11.10 [01:06:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.85, 1.85, 1.65 [01:07:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.84, 1.84 [01:08:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.61, 11.64, 11.19 [01:08:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.19, 1.56, 1.56 [01:13:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.82, 1.82, 1.83 [01:14:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.73, 11.45, 11.15 [01:16:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.84, 10.84, 10.94 [01:17:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.94, 2.24, 1.98 [01:18:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.48, 11.24, 11.06 [01:20:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.48, 11.46, 11.17 [01:22:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.65, 11.72, 11.29 [01:24:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.10, 11.42, 11.26 [01:26:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.35, 11.80, 11.42 [01:28:26] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.32, 1.87, 1.58 [01:32:15] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.22, 1.78, 1.50 [01:32:55] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.69, 3.49, 2.35 [01:34:17] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.95, 1.85, 1.72 [01:34:55] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.95, 3.70, 2.58 [01:36:15] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.14, 1.70, 1.69 [01:36:15] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.10, 1.67, 1.54 [01:38:55] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.99, 3.30, 2.67 [01:40:07] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.77, 2.08, 1.83 [01:42:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.70, 1.93, 1.81 [01:42:54] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:42:56] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:43:02] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:44:01] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.80, 1.54, 1.68 [01:44:53] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 4.590 second response time [01:44:54] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.650 second response time [01:45:02] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.988 second response time [01:47:53] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.61, 1.76, 1.74 [01:49:50] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.40, 1.62, 1.69 [01:54:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.15, 11.46, 11.95 [02:00:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [02:00:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.45, 9.73, 8.94 [02:00:42] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.71, 1.61, 1.49 [02:01:20] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.83, 5.69, 4.01 [02:02:23] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.88, 9.64, 8.99 [02:02:39] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.03, 1.44, 1.44 [02:03:15] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.80, 4.81, 3.87 [02:08:59] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 6.20, 6.94, 5.11 [02:11:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.56, 1.76, 1.99 [02:15:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.55, 10.38, 9.75 [02:17:04] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.59, 9.74, 9.60 [02:17:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.19, 2.20, 2.06 [02:20:31] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 4.70, 5.74, 5.70 [02:23:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.81, 1.62, 1.46 [02:25:01] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.16, 2.14, 1.69 [02:26:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.52, 11.24, 11.04 [02:26:31] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 4.01, 4.37, 5.06 [02:26:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.37, 1.86, 1.64 [02:27:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.45, 1.73, 1.94 [02:28:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.83, 11.14, 11.04 [02:28:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.20, 2.13, 1.77 [02:30:53] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.79, 1.99, 1.76 [02:32:31] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 20.51, 15.20, 9.19 [02:32:50] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.08, 1.67, 1.67 [02:43:31] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.33, 1.54, 1.70 [02:46:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.03, 9.08, 9.92 [02:49:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.58, 2.14, 1.90 [02:51:28] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.87, 1.90, 1.63 [02:55:23] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.14, 1.65, 1.60 [03:06:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.33, 10.40, 9.99 [03:08:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.91, 9.56, 9.72 [03:10:31] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 1.12, 3.84, 5.61 [03:11:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.93, 1.55, 1.43 [03:12:31] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 0.43, 2.70, 4.98 [03:13:52] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.46, 1.58, 1.46 [03:20:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.08, 1.76, 1.59 [03:24:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.27, 1.59, 1.57 [03:47:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.22, 10.27, 9.56 [03:50:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.86, 1.55, 1.36 [03:50:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.23, 1.83, 1.51 [03:51:17] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.83, 10.00, 9.60 [03:52:34] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.16, 2.04, 1.57 [03:52:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.49, 1.78, 1.53 [03:56:28] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.49, 1.92, 1.65 [03:56:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.56, 1.70, 1.56 [03:58:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.78, 1.56, 1.55 [04:03:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.47, 1.70, 1.98 [04:13:31] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.40, 1.42, 1.69 [04:19:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.92, 2.01, 1.82 [04:25:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.54, 1.92, 1.89 [04:26:46] PROBLEM - datawiki.online - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'datawiki.online' expires in 15 day(s) (Wed 03 Aug 2022 04:18:47 GMT +0000). [04:33:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/545666952ee4...e851fbfe6928 [04:33:26] [url] Comparing 545666952ee4...e851fbfe6928 · miraheze/ssl · GitHub | github.com [04:33:27] [02miraheze/ssl] 07MirahezeSSLBot 03e851fbf - Bot: Update SSL cert for datawiki.online [04:38:24] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.53, 1.98, 1.57 [04:40:22] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.70, 1.87, 1.58 [04:42:19] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.79, 1.50, 1.48 [04:45:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.42, 2.09, 1.92 [04:56:01] RECOVERY - datawiki.online - LetsEncrypt on sslhost is OK: OK - Certificate 'datawiki.online' will expire on Sun 16 Oct 2022 03:33:20 GMT +0000. [05:06:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.42, 10.22, 9.17 [05:08:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.01, 10.02, 9.21 [05:36:17] [02mediawiki] 07Universal-Omega labeled pull request 03#8159: Bump skins/Cosmos from `48e1b8a` to `3c3c184` - 13https://github.com/miraheze/mediawiki/pull/8159 [05:36:17] [url] Page not found · GitHub · GitHub | github.com [05:39:23] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.24, 1.80, 1.48 [05:39:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.57, 1.69, 1.96 [05:41:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.89, 1.50, 1.41 [05:47:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.36, 1.83, 1.87 [05:53:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.41, 2.00, 1.99 [06:05:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.11, 1.68, 1.75 [06:35:21] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 0.98, 2.08, 1.46 [06:37:15] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.51, 1.58, 1.36 [06:46:15] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.75, 1.52, 1.18 [06:48:15] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.69, 1.45, 1.19 [06:56:55] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 5.60, 3.82, 2.37 [07:01:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.47, 10.30, 9.69 [07:03:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.82, 10.11, 9.69 [07:07:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.30, 10.61, 10.03 [07:09:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.69, 9.84, 9.83 [07:10:55] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.69, 3.23, 3.42 [07:12:55] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.68, 2.71, 3.20 [07:16:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.70, 10.25, 10.13 [07:18:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.75, 10.15, 10.10 [08:25:38] RECOVERY - www.fortis.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fortis.wiki' will expire on Mon 22 Aug 2022 15:42:16 GMT +0000. [08:57:38] PROBLEM - www.fortis.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.fortis.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [09:06:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.76, 10.00, 9.11 [09:08:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.40, 9.57, 9.05 [09:08:43] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [09:08:48] PROBLEM - cp20 Stunnel HTTP for mwtask111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:28] PROBLEM - cp30 Stunnel HTTP for mwtask111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:51] PROBLEM - cp31 Stunnel HTTP for mwtask111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:10:19] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 19.30, 11.39, 5.72 [09:10:20] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:10:21] PROBLEM - cp21 Stunnel HTTP for mwtask111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:10:43] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [09:11:25] RECOVERY - cp30 Stunnel HTTP for mwtask111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15950 bytes in 0.322 second response time [09:11:47] RECOVERY - cp31 Stunnel HTTP for mwtask111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.315 second response time [09:12:18] RECOVERY - cp21 Stunnel HTTP for mwtask111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.030 second response time [09:12:19] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 0.472 second response time [09:12:51] RECOVERY - cp20 Stunnel HTTP for mwtask111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15956 bytes in 0.023 second response time [09:18:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.49, 10.17, 9.57 [09:19:52] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.71, 3.19, 4.00 [09:20:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.54, 9.48, 9.39 [09:23:42] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.06, 2.00, 3.33 [09:29:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.02, 1.40, 1.98 [09:47:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.12, 2.14, 1.92 [09:56:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.58, 3.79, 3.95 [09:59:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.30, 3.53, 4.00 [10:01:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.62, 3.88, 4.04 [10:01:20] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.67, 2.00, 1.52 [10:02:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.66, 3.96, 3.92 [10:03:14] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.89, 1.65, 1.45 [10:08:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.12, 3.82, 3.94 [10:10:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.49, 3.98, 3.97 [10:11:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.60, 1.87, 1.99 [10:15:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 1.92, 1.99, 2.01 [10:34:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.34, 3.28, 3.90 [10:40:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.16, 4.47, 4.16 [10:44:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.48, 3.55, 3.87 [10:46:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.63, 4.05, 4.02 [10:54:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.62, 3.58, 3.88 [10:56:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.33, 3.74, 3.89 [10:57:03] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.78, 1.85, 1.41 [10:58:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.81, 3.91 [10:59:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.56, 1.74, 1.99 [11:00:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.16, 3.83, 3.89 [11:00:51] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.26, 1.53, 1.37 [11:06:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.96, 3.41, 3.67 [11:09:42] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/81917ce5e6be...ce0c205c4285 [11:09:43] [url] Comparing 81917ce5e6be...ce0c205c4285 · miraheze/CreateWiki · GitHub | github.com [11:09:43] [02miraheze/CreateWiki] 07translatewiki 03ce0c205 - Localisation updates from https://translatewiki.net. [11:09:44] [url] Main page - translatewiki.net | translatewiki.net [11:09:45] [02miraheze/ErrorPages] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/ErrorPages/compare/4dac7f774b07...39acc1161114 [11:09:46] [url] Comparing 4dac7f774b07...39acc1161114 · miraheze/ErrorPages · GitHub | github.com [11:09:46] [02miraheze/ErrorPages] 07translatewiki 0339acc11 - Localisation updates from https://translatewiki.net. [11:09:47] [url] Main page - translatewiki.net | translatewiki.net [11:09:48] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/landing/compare/823411dd3171...9896519dc8b7 [11:09:48] [url] Comparing 823411dd3171...9896519dc8b7 · miraheze/landing · GitHub | github.com [11:09:49] [02miraheze/landing] 07translatewiki 039896519 - Localisation updates from https://translatewiki.net. [11:09:50] [url] Main page - translatewiki.net | translatewiki.net [11:09:51] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/f7eb58a7e4a5...1987f84656ef [11:09:51] [url] Comparing f7eb58a7e4a5...1987f84656ef · miraheze/MirahezeMagic · GitHub | github.com [11:09:52] [02miraheze/MirahezeMagic] 07translatewiki 031987f84 - Localisation updates from https://translatewiki.net. [11:09:53] [url] Main page - translatewiki.net | translatewiki.net [11:10:10] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.21, 3.65, 3.68 [11:11:11] miraheze/landing - translatewiki the build passed. [11:13:31] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.53, 1.48, 1.68 [11:18:39] miraheze/CreateWiki - translatewiki the build passed. [11:18:56] PROBLEM - fortis.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - fortis.wiki All nameservers failed to answer the query. [11:19:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.76, 2.26, 1.94 [11:21:23] miraheze/MirahezeMagic - translatewiki the build passed. [11:21:49] RECOVERY - fortis.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fortis.wiki' will expire on Mon 22 Aug 2022 15:42:16 GMT +0000. [11:25:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.48, 1.97, 1.96 [11:37:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.09, 1.89, 1.89 [11:39:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.95, 1.92, 1.91 [11:41:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.41, 2.07, 1.96 [11:48:53] PROBLEM - fortis.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fortis.wiki could not be found [11:53:46] PROBLEM - fortis.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address fortis.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [12:27:38] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.71, 1.54, 1.27 [12:29:32] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.18, 1.75, 1.38 [12:31:26] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.82, 1.68, 1.39 [12:35:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.69, 11.52, 10.39 [12:35:15] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.22, 1.91, 1.54 [12:35:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.71, 9.49, 8.86 [12:37:09] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.35, 1.69, 1.50 [12:37:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.94, 10.93, 10.31 [12:37:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.30, 9.15, 8.80 [12:39:06] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.31, 10.10, 10.08 [12:40:59] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.38, 1.74, 1.58 [12:41:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.31, 1.71, 2.00 [12:42:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.73, 9.65, 8.41 [12:42:53] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.85, 1.47, 1.50 [12:44:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.62, 9.34, 8.46 [12:47:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.97, 2.06, 2.01 [12:49:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.37, 10.08, 9.96 [12:53:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.39, 11.36, 10.52 [12:57:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.98, 11.15, 10.62 [12:59:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.41, 1.72, 1.93 [13:00:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [13:03:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.05, 1.84, 1.93 [13:05:19] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.68, 10.65, 10.45 [13:05:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.96, 1.86, 1.92 [13:05:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [13:07:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.87, 10.35, 10.37 [13:09:12] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.07, 9.86, 10.18 [13:17:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.18, 1.89, 1.85 [13:27:10] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.88, 1.62, 1.36 [13:30:59] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.30, 1.99, 1.56 [13:32:54] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.58, 1.84, 1.55 [13:34:48] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.00, 1.54, 1.47 [13:39:54] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 5.66, 4.01, 2.15 [13:42:31] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [13:47:45] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.50, 1.59, 1.71 [13:49:45] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.21, 1.44, 1.63 [13:54:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [13:56:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.92, 10.68, 9.88 [14:00:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.90, 9.81, 9.72 [14:05:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.64, 10.28, 9.47 [14:07:11] PROBLEM - wiki.fossbots.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fossbots.org' expires in 15 day(s) (Wed 03 Aug 2022 13:59:56 GMT +0000). [14:07:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.21, 9.95, 9.45 [14:09:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [14:10:31] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:11:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e851fbfe6928...4c98ee6f176a [14:11:49] [url] Comparing e851fbfe6928...4c98ee6f176a · miraheze/ssl · GitHub | github.com [14:11:50] [02miraheze/ssl] 07MirahezeSSLBot 034c98ee6 - Bot: Update SSL cert for wiki.fossbots.org [14:19:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.56, 10.84, 10.11 [14:25:41] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.93, 9.87, 9.96 [14:31:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.28, 1.64, 2.00 [14:36:21] RECOVERY - wiki.fossbots.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fossbots.org' will expire on Sun 16 Oct 2022 13:11:42 GMT +0000. [14:36:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.37, 11.01, 10.43 [14:42:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.07, 10.79, 9.21 [14:44:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.95, 9.63, 8.98 [14:44:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.31, 9.65, 9.01 [14:46:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.33, 9.70, 10.08 [14:46:24] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.76, 9.06, 8.86 [14:47:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.21, 2.13, 1.91 [14:50:55] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.91, 3.41, 2.78 [14:52:55] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.13, 3.19, 2.77 [14:54:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.52, 10.62, 10.32 [14:58:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.61, 10.55, 9.84 [14:58:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.30, 10.14, 10.18 [14:59:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.93, 2.00 [15:00:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.45, 10.74, 10.00 [15:04:08] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.56, 11.33, 10.33 [15:06:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.73, 11.25, 10.44 [15:11:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.54, 2.18, 2.04 [15:13:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.88, 9.63, 10.00 [15:21:40] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.19, 1.61, 1.22 [15:23:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.02, 1.43, 1.20 [15:24:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.91, 11.27, 10.25 [15:24:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.05, 10.40, 10.23 [15:25:37] RECOVERY - www.fortis.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fortis.wiki' will expire on Mon 22 Aug 2022 15:42:16 GMT +0000. [15:26:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.83, 10.96, 10.26 [15:27:16] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.26, 9.85, 8.60 [15:28:00] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.70, 11.40, 10.49 [15:28:39] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.11, 11.69, 10.77 [15:29:16] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.01, 9.53, 8.64 [15:29:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.79, 10.41, 9.48 [15:29:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.08, 11.36, 10.59 [15:31:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.71, 10.43, 9.85 [15:32:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.77, 10.37, 9.89 [15:32:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.98, 11.39, 10.88 [15:33:14] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.52, 10.10, 9.80 [15:34:12] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.64, 10.15, 9.86 [15:34:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.45, 12.40, 11.30 [15:35:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.08, 11.68, 10.95 [15:36:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.15, 11.95, 11.28 [15:37:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.47, 11.47, 10.97 [15:37:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.03, 9.57, 9.53 [15:42:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.70, 11.74, 11.34 [15:44:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.36, 11.66, 11.36 [15:47:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [15:57:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [15:57:20] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.43, 1.77, 1.41 [15:57:34] PROBLEM - www.fortis.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.fortis.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:59:14] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.16, 1.85, 1.47 [16:00:56] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.99, 9.57, 10.09 [16:01:08] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.55, 1.75, 1.48 [16:03:03] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.41, 1.60, 1.45 [16:04:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.90, 11.02, 10.77 [16:05:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.77, 10.69, 10.69 [16:06:48] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.11, 11.39, 10.69 [16:08:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.76, 10.73, 10.54 [16:10:40] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.83, 9.67, 10.17 [16:11:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.75, 1.80, 1.99 [16:11:53] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.71, 9.37, 10.13 [16:17:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.26, 1.81, 1.90 [16:19:42] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.41, 10.47, 10.26 [16:20:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.70, 10.69, 10.23 [16:21:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.72, 10.07, 10.13 [16:22:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.84, 11.21, 10.46 [16:24:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.53, 11.24, 10.57 [16:27:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.22, 1.81, 1.98 [16:28:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.94, 9.86, 9.03 [16:29:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.15, 11.15, 10.55 [16:30:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.77, 9.65, 9.04 [16:31:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.15, 10.64, 10.45 [16:32:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.55, 11.26, 9.97 [16:32:37] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.96, 2.03, 1.64 [16:33:59] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.56, 11.17, 10.08 [16:34:32] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.72, 1.95, 1.66 [16:36:30] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.15, 1.65, 1.58 [16:41:57] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.62, 9.84, 9.89 [16:42:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.17, 9.33, 10.10 [16:43:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.13, 9.27, 10.03 [16:44:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [16:47:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.98, 2.16, 1.95 [17:07:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.85, 10.28, 9.79 [17:09:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.02, 9.49, 9.56 [17:14:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [17:15:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.74, 9.78, 8.97 [17:17:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 20.46, 12.77, 10.10 [17:19:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.89, 10.86, 10.15 [17:19:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.41, 3.83, 2.34 [17:21:07] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.16, 11.22, 10.13 [17:21:27] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 16.94, 12.79, 10.91 [17:21:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.82, 9.82, 9.12 [17:21:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.14, 11.07, 10.06 [17:25:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.71, 10.91, 10.29 [17:25:45] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.66, 9.74, 9.27 [17:27:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.21, 9.87, 9.92 [17:30:57] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.48, 11.37, 10.62 [17:31:43] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 12.00, 11.04, 9.93 [17:33:43] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.31, 11.59, 10.28 [17:35:34] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.62, 1.76, 1.96 [17:35:42] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.66, 11.04, 10.22 [17:38:08] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:38:22] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:38:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.36, 11.54, 11.14 [17:39:41] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.78, 9.68, 9.88 [17:40:08] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.975 second response time [17:40:22] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28480 bytes in 5.286 second response time [17:41:34] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.52, 11.14, 9.95 [17:43:23] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.83, 1.30, 1.67 [17:43:29] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.66, 11.28, 10.17 [17:44:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [17:45:39] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.35, 10.43, 10.16 [17:46:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.17, 11.87, 11.96 [17:46:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.32, 11.72, 11.24 [17:47:39] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.94, 10.18, 10.10 [17:48:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.75, 12.61, 12.22 [17:48:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.65, 11.31, 11.15 [17:49:13] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.38, 10.01, 9.93 [17:51:49] RECOVERY - fortis.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fortis.wiki' will expire on Mon 22 Aug 2022 15:42:16 GMT +0000. [17:56:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.49, 11.17, 11.79 [17:58:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.88, 9.31, 10.19 [18:00:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.21, 12.09, 12.02 [18:02:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.61, 11.49, 11.80 [18:02:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.22, 10.65, 10.51 [18:03:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.61, 1.66, 1.98 [18:04:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [18:04:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.34, 10.55, 10.50 [18:05:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.39, 1.77, 1.97 [18:06:15] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.53, 2.93, 1.99 [18:06:17] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.99, 9.57, 10.14 [18:06:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.08, 11.83, 11.83 [18:08:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.43, 11.50, 11.70 [18:10:15] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.65, 1.74, 1.72 [18:12:08] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.96, 10.24, 10.21 [18:14:15] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.30, 1.62, 1.68 [18:18:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.33, 10.78, 10.40 [18:18:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.39, 11.38, 11.39 [18:18:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.11, 10.11, 9.56 [18:20:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.73, 10.59, 9.71 [18:20:29] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.12, 10.59, 9.79 [18:21:26] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.98, 10.54, 8.98 [18:21:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.58, 10.90, 10.57 [18:22:28] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.88, 11.10, 10.08 [18:22:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.75, 9.85, 9.27 [18:23:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.22, 10.60, 9.19 [18:23:41] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:44] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:46] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:49] PROBLEM - fortis.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address fortis.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:23:56] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.39, 11.25, 10.72 [18:24:28] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.44, 11.73, 10.43 [18:24:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.74, 9.66, 9.28 [18:25:04] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:25:43] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:26:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.65, 9.38, 9.49 [18:26:27] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.13, 10.26, 10.05 [18:27:25] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.46, 9.79, 9.19 [18:27:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.27, 11.29, 10.88 [18:29:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.17, 11.65, 11.06 [18:30:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.21, 10.28, 9.79 [18:30:26] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.01, 11.03, 10.39 [18:31:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.90, 11.28, 11.00 [18:32:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.64, 10.45, 9.92 [18:32:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.56, 10.91, 10.43 [18:34:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 3.83, 8.29, 9.22 [18:34:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 4.56, 9.65, 11.28 [18:34:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 4.54, 8.33, 9.53 [18:35:01] [discord] Having 40+ sec response times... and 502 [18:35:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [18:35:05] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:06] [discord] even meta is very slow [18:35:13] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:14] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:16] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.906 second response time [18:35:29] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:32] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:32] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:42] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:43] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.43, 8.52, 9.90 [18:35:44] [discord] looks like the hiccup has passed [18:35:47] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:52] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:58] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.653 second response time [18:36:00] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.911 second response time [18:36:00] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 4.020 second response time [18:36:07] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:08] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:14] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.332 second response time [18:36:14] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:15] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw122 [18:36:20] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:34] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:34] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:35] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:37] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:39] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:39] [discord] and everything is back to normal [18:36:47] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:02] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:04] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw122 [18:37:19] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.608 second response time [18:37:22] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [18:37:31] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.918 second response time [18:37:38] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.754 second response time [18:38:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 3.47, 6.96, 9.87 [18:40:15] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [18:40:38] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.89, 6.48, 4.40 [18:41:04] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [18:41:09] [discord] extension stylesheets still take like 30 sec and now it's back to 502 and not loading [18:41:11] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 9.703 second response time [18:41:17] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.933 second response time [18:41:23] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [18:41:27] [discord] and now it loads but still freezing on styles... [18:41:40] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.729 second response time [18:41:41] [discord] idk, just very unstable [18:41:44] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.022 second response time [18:41:45] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.990 second response time [18:41:56] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.024 second response time [18:41:58] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.385 second response time [18:42:19] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 4.515 second response time [18:42:20] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 6.130 second response time [18:42:23] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 1.762 second response time [18:42:27] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.797 second response time [18:42:36] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 2.32, 4.99, 4.11 [18:42:43] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.170 second response time [18:42:44] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.483 second response time [18:42:49] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.343 second response time [18:42:49] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.64, 3.97, 2.47 [18:42:53] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.465 second response time [18:42:53] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.319 second response time [18:43:00] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [18:46:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.81, 10.23, 10.00 [18:48:41] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.80, 1.69, 1.88 [18:50:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.09, 9.07, 8.23 [18:50:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.40, 11.17, 10.48 [18:52:17] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.13, 8.93, 8.27 [18:54:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.01, 10.19, 10.20 [18:54:33] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.24, 1.41, 1.68 [18:58:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.02, 10.34, 10.28 [19:00:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.36, 1.82, 1.42 [19:02:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.77, 1.84, 1.47 [19:04:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.68, 11.36, 10.73 [19:06:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.56, 10.54, 9.57 [19:06:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.41, 11.58, 10.90 [19:06:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.50, 2.57, 1.82 [19:08:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.65, 9.68, 9.37 [19:11:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.35, 9.92, 8.75 [19:12:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.41, 1.88, 1.77 [19:14:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.50, 10.18, 9.66 [19:14:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.14, 1.58, 1.67 [19:15:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.98, 10.11, 9.12 [19:20:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.42, 10.33, 9.54 [19:22:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.89, 9.81, 9.45 [19:23:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.25, 11.33, 10.50 [19:24:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.50, 11.65, 11.14 [19:24:40] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:24:55] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:25:06] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:25:07] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:25:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.46, 10.13, 9.19 [19:25:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.35, 11.19, 10.53 [19:26:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.84, 11.63, 11.19 [19:26:39] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.217 second response time [19:26:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.61, 10.34, 9.64 [19:26:56] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.364 second response time [19:27:06] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.413 second response time [19:27:07] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 4.730 second response time [19:27:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.73, 9.51, 9.08 [19:28:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.78, 9.89, 9.57 [19:32:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.14, 11.48, 11.22 [19:32:58] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:19] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:23] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:24] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.43, 10.68, 10.46 [19:33:45] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.48, 1.65, 1.09 [19:34:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.47, 11.37, 11.21 [19:34:30] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.46, 2.31, 1.53 [19:35:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.52, 10.22, 10.29 [19:35:45] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.78, 1.80, 1.22 [19:36:30] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.50, 1.96, 1.50 [19:37:07] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.398 second response time [19:37:45] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.95, 1.42, 1.14 [19:38:30] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.74, 1.59, 1.42 [19:39:33] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 5.593 second response time [19:39:34] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.008 second response time [19:39:36] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.793 second response time [19:41:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.65, 11.18, 10.66 [19:41:59] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.24, 10.47, 9.66 [19:42:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.22, 10.93, 9.89 [19:43:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.25, 10.66, 10.56 [19:43:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.00, 10.21, 9.67 [19:44:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.08, 10.41, 9.86 [19:45:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 10.16, 9.95, 9.62 [19:46:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.97, 10.03, 9.77 [19:50:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.46, 10.60, 10.15 [19:50:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.83, 11.41, 11.23 [19:54:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.64, 9.17, 9.74 [19:54:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.67, 10.94, 11.17 [19:54:31] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 4.13, 4.50, 2.62 [19:57:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.70, 9.65, 10.15 [19:58:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.10, 11.69, 11.44 [20:00:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.15, 10.83, 10.05 [20:01:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.56, 10.05, 10.21 [20:02:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.54, 10.70, 10.08 [20:02:30] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.04, 1.79, 2.00 [20:03:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.95, 11.12, 10.59 [20:03:59] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.86, 10.14, 9.96 [20:07:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.32, 10.85, 10.66 [20:08:31] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.26, 1.00, 1.59 [20:10:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.18, 11.89, 11.79 [20:17:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.04, 9.62, 10.18 [20:17:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.86, 11.24, 10.18 [20:20:02] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [20:21:33] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.88, 10.13, 9.94 [20:24:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.92, 8.69, 9.97 [20:28:30] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 4.89, 2.29, 1.52 [20:30:30] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.22, 1.70, 1.39 [20:32:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.63, 9.77, 9.24 [20:34:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.24, 9.64, 9.27 [20:36:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.47, 10.94, 10.11 [20:40:29] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.49, 10.12, 9.97 [20:40:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.70, 10.23, 9.68 [20:44:32] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.56, 9.90, 9.70 [20:49:58] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.40, 9.98, 9.79 [20:51:05] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:23] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:32] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [20:51:36] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:40] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:52:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.83, 10.76, 10.08 [20:54:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.64, 10.55, 10.25 [20:54:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.20, 9.86, 9.85 [20:55:11] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.820 second response time [20:55:32] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.878 second response time [20:55:41] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.485 second response time [20:55:42] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 2.898 second response time [20:56:04] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.060 second response time [21:11:01] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.44, 1.65, 1.32 [21:12:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.07, 1.37, 1.25 [21:17:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.51, 9.68, 9.54 [21:19:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.69, 9.39, 9.44 [21:21:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.91, 11.73, 11.02 [21:21:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.24, 9.76, 9.31 [21:23:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.52, 11.10, 10.87 [21:24:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.18, 1.82, 1.49 [21:25:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.47, 12.00, 11.22 [21:25:36] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.03, 9.64, 9.38 [21:26:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.36, 1.62, 1.45 [21:28:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.54, 11.16, 11.06 [21:31:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.67, 10.48, 9.79 [21:32:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.86, 10.23, 9.65 [21:32:48] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.23, 11.46, 11.17 [21:33:33] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.30, 10.56, 9.86 [21:33:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.17, 11.18, 10.12 [21:34:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.32, 9.62, 9.50 [21:34:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.68, 11.25, 11.11 [21:35:33] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.24, 10.08, 9.77 [21:35:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.70, 11.69, 10.46 [21:36:40] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.52, 11.94, 11.37 [21:37:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.29, 12.02, 10.71 [21:38:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.94, 10.85, 11.04 [21:39:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.39, 11.69, 10.75 [21:43:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.05, 12.19, 11.13 [21:44:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.36, 10.45, 9.82 [21:45:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.72, 11.96, 11.17 [21:46:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.53, 10.55, 9.98 [21:46:32] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [21:47:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.99, 12.04, 11.31 [21:48:22] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.72, 1.74, 1.31 [21:49:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.91, 11.26, 11.11 [21:50:19] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.63, 1.62, 1.31 [21:52:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.81, 11.41, 11.02 [21:54:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.84, 11.07, 10.94 [22:00:26] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.03, 11.11, 10.59 [22:02:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.87, 11.10, 10.66 [22:05:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.16, 11.28, 10.96 [22:06:24] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.12, 11.30, 10.83 [22:07:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.06, 11.28, 11.01 [22:08:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.32, 9.63, 10.07 [22:08:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.82, 11.56, 10.98 [22:10:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.98, 9.98, 9.39 [22:12:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.39, 10.82, 10.45 [22:12:31] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [22:14:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.28, 10.54, 10.39 [22:14:32] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.13, 9.87, 9.51 [22:15:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.06, 10.97, 10.90 [22:17:02] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [22:17:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.52, 11.03, 10.95 [22:18:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.61, 9.82, 10.16 [22:22:15] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.85, 1.42, 1.17 [22:22:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.35, 11.18, 10.64 [22:22:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.70, 1.77, 1.37 [22:24:15] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.34, 1.43, 1.21 [22:24:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.24, 10.74, 10.54 [22:24:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.09, 1.92, 1.47 [22:26:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.68, 9.55, 10.04 [22:26:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.90, 1.64, 1.43 [22:32:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.24, 10.42, 10.26 [22:34:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.67, 9.61, 10.00 [22:34:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.52, 11.35, 10.76 [22:35:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.18, 11.44, 10.90 [22:36:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.83, 11.20, 10.76 [22:39:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.01, 10.95, 10.86 [22:40:16] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.57, 9.55, 10.14 [22:40:31] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:47:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.91, 9.49, 10.19 [22:50:59] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.21, 1.91, 1.33 [22:51:28] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:52:38] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.010 second response time [22:52:41] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [22:52:56] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.77, 1.43, 1.22 [22:53:06] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:53:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.52, 9.89, 8.96 [22:53:15] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.07, 2.55, 1.82 [22:53:28] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:54:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.20, 9.55, 10.16 [22:54:38] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.079 second response time [22:54:40] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.194 second response time [22:55:07] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 5.534 second response time [22:55:12] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.31, 2.00, 1.70 [22:55:25] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.330 second response time [22:55:33] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.627 second response time [22:57:08] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.70, 9.44, 9.03 [22:57:09] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.83, 1.62, 1.60 [22:58:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.44, 10.34, 10.37 [23:00:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.30, 11.42, 10.76 [23:00:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.82, 10.60, 10.33 [23:02:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.43, 11.60, 10.93 [23:04:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.53, 11.72, 11.04 [23:04:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.44, 11.87, 10.85 [23:05:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.20, 10.30, 9.74 [23:05:10] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.60, 10.89, 10.17 [23:06:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.53, 11.87, 11.00 [23:06:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.06, 9.92, 9.68 [23:07:09] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.23, 9.77, 9.84 [23:10:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.77, 12.22, 11.30 [23:11:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.64, 1.77, 1.99 [23:11:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.62, 10.49, 9.66 [23:13:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.66, 10.22, 9.68 [23:14:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.12, 11.79, 11.54 [23:14:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.43, 10.80, 10.25 [23:16:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.06, 11.86, 11.59 [23:16:30] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.36, 10.14, 10.07 [23:17:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.17, 2.09, 2.00 [23:18:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.97, 11.47, 11.39 [23:18:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.47, 11.70, 11.57 [23:19:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.87, 9.23, 9.51 [23:20:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.07, 11.61, 11.53 [23:24:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.72, 11.76, 11.62 [23:27:35] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:28:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.95, 11.95, 11.68 [23:30:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.61, 11.94, 11.72 [23:31:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.83, 1.87, 1.96 [23:31:41] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 5.844 second response time [23:32:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.07, 12.02, 11.78 [23:33:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.07, 1.92, 1.97 [23:34:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.44, 11.91, 11.78 [23:35:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.46, 1.77, 1.91 [23:36:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.88, 11.16, 10.32 [23:36:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.22, 12.34, 11.95 [23:37:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.98, 10.34, 10.04 [23:38:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.91, 11.52, 10.56 [23:39:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.90, 1.93 [23:39:33] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.85, 10.11, 9.99 [23:41:31] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.58, 1.83, 1.91 [23:43:24] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.56, 10.70, 10.28 [23:43:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.61, 10.35, 9.92 [23:44:20] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.63, 11.65, 11.80 [23:45:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.89, 11.37, 10.57 [23:47:13] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.45, 10.24, 10.25 [23:47:24] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.78, 9.82, 9.83 [23:47:31] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.17, 2.20, 2.02 [23:47:58] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.39, 9.10, 9.92 [23:51:03] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.66, 9.45, 9.92 [23:51:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.53, 9.41, 10.20 [23:53:56] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.51, 10.42, 10.25 [23:55:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.07, 10.25, 10.35