[00:01:01] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.21, 1.87, 1.80 [00:01:07] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.79, 1.77, 1.57 [00:02:55] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.76, 2.01, 1.87 [00:03:03] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.09, 1.61, 1.54 [00:03:29] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:42] PROBLEM - es131 Puppet on es131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [00:04:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.26, 1.76, 1.79 [00:04:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.63, 11.84, 11.16 [00:05:13] PROBLEM - es101 Puppet on es101 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [00:05:19] PROBLEM - es121 Puppet on es121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [00:05:19] PROBLEM - es141 Puppet on es141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [00:05:30] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 1.678 second response time [00:05:36] PROBLEM - es111 Puppet on es111 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [00:06:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.27, 11.64, 11.18 [00:08:26] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.94, 3.26, 2.34 [00:10:23] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 2.96, 3.18, 2.42 [00:10:44] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.21, 11.96, 11.39 [00:12:27] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.14, 1.41, 1.64 [00:12:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.03, 11.67, 11.36 [00:14:41] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.71, 2.10, 1.65 [00:14:50] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.47, 10.94, 9.80 [00:16:37] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.17, 1.75, 1.58 [00:16:50] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.69, 10.58, 9.82 [00:20:32] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.43, 1.87, 1.64 [00:20:50] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.05, 9.28, 9.51 [00:21:59] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.72, 1.46, 1.51 [00:22:14] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.91, 2.00 [00:22:32] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.91, 1.77, 1.63 [00:23:54] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.19, 1.43, 1.50 [00:24:32] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.33, 1.55, 1.56 [00:28:08] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:33] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:41] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:30:13] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.910 second response time [00:30:38] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.226 second response time [00:30:48] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.944 second response time [00:32:09] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [00:32:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Translate-ffbe9ee [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4d5e4fe9dc4b [00:32:59] [02miraheze/mediawiki] 07dependabot[bot] 034d5e4fe - Bump extensions/Translate from `d7618b5` to `ffbe9ee` [00:33:01] [02mediawiki] 07dependabot[bot] opened pull request 03#8317: Bump extensions/Translate from `d7618b5` to `ffbe9ee` - 13https://github.com/miraheze/mediawiki/pull/8317 [00:33:01] [url] Page not found · GitHub · GitHub | github.com [00:33:02] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Translate-ffbe9ee - 13https://github.com/miraheze/mediawiki [00:33:02] ... [00:33:04] [02mediawiki] 07dependabot[bot] commented on pull request 03#8310: Bump extensions/Translate from `d7618b5` to `9366a0a` - 13https://github.com/miraheze/mediawiki/pull/8310#issuecomment-1200546264 [00:33:04] [url] Bump extensions/Translate from `d7618b5` to `9366a0a` by dependabot[bot] · Pull Request #8310 · miraheze/mediawiki · GitHub | github.com [00:33:05] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Echo-c713726 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0dc1771b79b6 [00:33:07] [02miraheze/mediawiki] 07dependabot[bot] 030dc1771 - Bump extensions/Echo from `bc2a368` to `c713726` [00:33:08] [02mediawiki] 07dependabot[bot] closed pull request 03#8310: Bump extensions/Translate from `d7618b5` to `9366a0a` - 13https://github.com/miraheze/mediawiki/pull/8310 [00:33:08] [url] Page not found · GitHub · GitHub | github.com [00:33:10] [02mediawiki] 07dependabot[bot] opened pull request 03#8318: Bump extensions/Echo from `bc2a368` to `c713726` - 13https://github.com/miraheze/mediawiki/pull/8318 [00:33:10] [url] Page not found · GitHub · GitHub | github.com [00:33:11] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Translate-9366a0a [00:33:13] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Translate-9366a0a - 13https://github.com/miraheze/mediawiki [00:33:13] ... [00:33:14] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Echo-c713726 - 13https://github.com/miraheze/mediawiki [00:33:14] [url] Page not found · GitHub · GitHub | github.com [00:33:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/CiteThisPage-e1c68e0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/67a57538168f [00:33:17] [02miraheze/mediawiki] 07dependabot[bot] 0367a5753 - Bump extensions/CiteThisPage from `6c95f2f` to `e1c68e0` [00:33:19] [02mediawiki] 07dependabot[bot] opened pull request 03#8319: Bump extensions/CiteThisPage from `6c95f2f` to `e1c68e0` - 13https://github.com/miraheze/mediawiki/pull/8319 [00:33:19] ... [00:33:20] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/CiteThisPage-e1c68e0 - 13https://github.com/miraheze/mediawiki [00:33:20] [url] Page not found · GitHub · GitHub | github.com [00:33:22] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/WikibaseQualityConstraints-852f7a9 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c5f9d44a190a [00:33:23] [02miraheze/mediawiki] 07dependabot[bot] 03c5f9d44 - Bump extensions/WikibaseQualityConstraints from `862650e` to `852f7a9` [00:33:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/WikibaseQualityConstraints-852f7a9 - 13https://github.com/miraheze/mediawiki [00:33:25] [url] Page not found · GitHub · GitHub | github.com [00:33:26] [02mediawiki] 07dependabot[bot] opened pull request 03#8320: Bump extensions/WikibaseQualityConstraints from `862650e` to `852f7a9` - 13https://github.com/miraheze/mediawiki/pull/8320 [00:33:26] ... [00:33:28] [02mediawiki] 07dependabot[bot] opened pull request 03#8321: Bump extensions/ApprovedRevs from `d4b29d5` to `2c74834` - 13https://github.com/miraheze/mediawiki/pull/8321 [00:33:29] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ApprovedRevs-2c74834 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/13881e9c2a61 [00:33:31] [02miraheze/mediawiki] 07dependabot[bot] 0313881e9 - Bump extensions/ApprovedRevs from `d4b29d5` to `2c74834` [00:33:32] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ApprovedRevs-2c74834 - 13https://github.com/miraheze/mediawiki [00:33:34] [02mediawiki] 07dependabot[bot] commented on pull request 03#8117: Bump extensions/ApprovedRevs from `d4b29d5` to `87d4512` - 13https://github.com/miraheze/mediawiki/pull/8117#issuecomment-1200546336 [00:33:34] [url] Bump extensions/ApprovedRevs from `d4b29d5` to `87d4512` by dependabot[bot] · Pull Request #8117 · miraheze/mediawiki · GitHub | github.com [00:33:35] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/UploadsLink-bc21447 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/397fb2ee69f9 [00:33:37] [02miraheze/mediawiki] 07dependabot[bot] 03397fb2e - Bump extensions/UploadsLink from `bb40cef` to `bc21447` [00:33:38] [02mediawiki] 07dependabot[bot] opened pull request 03#8322: Bump extensions/UploadsLink from `bb40cef` to `bc21447` - 13https://github.com/miraheze/mediawiki/pull/8322 [00:33:39] [url] Page not found · GitHub · GitHub | github.com [00:33:40] [02mediawiki] 07dependabot[bot] closed pull request 03#8117: Bump extensions/ApprovedRevs from `d4b29d5` to `87d4512` - 13https://github.com/miraheze/mediawiki/pull/8117 [00:33:40] [url] Page not found · GitHub · GitHub | github.com [00:33:41] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/ApprovedRevs-87d4512 [00:33:43] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/UploadsLink-bc21447 - 13https://github.com/miraheze/mediawiki [00:33:43] ... [00:33:44] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/ApprovedRevs-87d4512 - 13https://github.com/miraheze/mediawiki [00:33:44] [url] Page not found · GitHub · GitHub | github.com [00:33:46] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Newsletter-1a9aaf9 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1acf12188570 [00:33:47] [02miraheze/mediawiki] 07dependabot[bot] 031acf121 - Bump extensions/Newsletter from `8e8e2ff` to `1a9aaf9` [00:33:49] [02mediawiki] 07dependabot[bot] opened pull request 03#8323: Bump extensions/Newsletter from `8e8e2ff` to `1a9aaf9` - 13https://github.com/miraheze/mediawiki/pull/8323 [00:33:49] ... [00:33:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Newsletter-1a9aaf9 - 13https://github.com/miraheze/mediawiki [00:33:50] [url] Page not found · GitHub · GitHub | github.com [00:33:52] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/OAuth-0c59a55 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/3e8ec55f76b7 [00:33:53] [02miraheze/mediawiki] 07dependabot[bot] 033e8ec55 - Bump extensions/OAuth from `f7c2faf` to `0c59a55` [00:33:55] [02mediawiki] 07dependabot[bot] opened pull request 03#8324: Bump extensions/OAuth from `f7c2faf` to `0c59a55` - 13https://github.com/miraheze/mediawiki/pull/8324 [00:33:55] [url] Page not found · GitHub · GitHub | github.com [00:33:56] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/OAuth-0c59a55 - 13https://github.com/miraheze/mediawiki [00:33:56] ... [00:33:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/TimedMediaHandler-07e93c0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/6371f5fa0359 [00:33:59] [02miraheze/mediawiki] 07dependabot[bot] 036371f5f - Bump extensions/TimedMediaHandler from `d7e650a` to `07e93c0` [00:34:01] [02mediawiki] 07dependabot[bot] opened pull request 03#8325: Bump extensions/TimedMediaHandler from `d7e650a` to `07e93c0` - 13https://github.com/miraheze/mediawiki/pull/8325 [00:34:02] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/TimedMediaHandler-07e93c0 - 13https://github.com/miraheze/mediawiki [00:34:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Evelution-fbe9301 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/740d8c9090f5 [00:34:05] [02miraheze/mediawiki] 07dependabot[bot] 03740d8c9 - Bump skins/Evelution from `631cf5e` to `fbe9301` [00:34:07] [02mediawiki] 07dependabot[bot] opened pull request 03#8326: Bump skins/Evelution from `631cf5e` to `fbe9301` - 13https://github.com/miraheze/mediawiki/pull/8326 [00:34:08] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Evelution-fbe9301 - 13https://github.com/miraheze/mediawiki [00:34:10] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/FlaggedRevs-92456b9 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e1bcd3fab6fd [00:34:11] [02miraheze/mediawiki] 07dependabot[bot] 03e1bcd3f - Bump extensions/FlaggedRevs from `4bde537` to `92456b9` [00:34:13] [02mediawiki] 07dependabot[bot] commented on pull request 03#8312: Bump skins/Evelution from `631cf5e` to `d2c24db` - 13https://github.com/miraheze/mediawiki/pull/8312#issuecomment-1200546465 [00:34:13] [url] Bump skins/Evelution from `631cf5e` to `d2c24db` by dependabot[bot] · Pull Request #8312 · miraheze/mediawiki · GitHub | github.com [00:34:14] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.19, 1.43, 1.70 [00:34:14] [02mediawiki] 07dependabot[bot] opened pull request 03#8327: Bump extensions/FlaggedRevs from `4bde537` to `92456b9` - 13https://github.com/miraheze/mediawiki/pull/8327 [00:34:14] [url] Page not found · GitHub · GitHub | github.com [00:34:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/FlaggedRevs-92456b9 - 13https://github.com/miraheze/mediawiki [00:34:16] [url] Page not found · GitHub · GitHub | github.com [00:34:17] [02mediawiki] 07dependabot[bot] closed pull request 03#8312: Bump skins/Evelution from `631cf5e` to `d2c24db` - 13https://github.com/miraheze/mediawiki/pull/8312 [00:34:18] ... [00:34:19] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Evelution-d2c24db [00:34:20] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Evelution-d2c24db - 13https://github.com/miraheze/mediawiki [00:34:20] [url] Page not found · GitHub · GitHub | github.com [00:34:22] [02mediawiki] 07dependabot[bot] commented on pull request 03#8201: Bump extensions/FlaggedRevs from `4bde537` to `f8f827b` - 13https://github.com/miraheze/mediawiki/pull/8201#issuecomment-1200546481 [00:34:22] [url] Bump extensions/FlaggedRevs from `4bde537` to `f8f827b` by dependabot[bot] · Pull Request #8201 · miraheze/mediawiki · GitHub | github.com [00:34:23] [02mediawiki] 07github-actions[bot] labeled pull request 03#8319: Bump extensions/CiteThisPage from `6c95f2f` to `e1c68e0` - 13https://github.com/miraheze/mediawiki/pull/8319 [00:34:23] [url] Page not found · GitHub · GitHub | github.com [00:34:25] [02mediawiki] 07dependabot[bot] closed pull request 03#8201: Bump extensions/FlaggedRevs from `4bde537` to `f8f827b` - 13https://github.com/miraheze/mediawiki/pull/8201 [00:34:25] [url] Page not found · GitHub · GitHub | github.com [00:34:26] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/FlaggedRevs-f8f827b [00:34:28] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/FlaggedRevs-f8f827b - 13https://github.com/miraheze/mediawiki [00:34:28] ... [00:34:29] [02mediawiki] 07dependabot[bot] opened pull request 03#8328: Bump extensions/RevisionSlider from `94b62eb` to `07b46fd` - 13https://github.com/miraheze/mediawiki/pull/8328 [00:34:30] ... [00:34:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/RevisionSlider-07b46fd [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1e6706d76062 [00:34:32] [02miraheze/mediawiki] 07dependabot[bot] 031e6706d - Bump extensions/RevisionSlider from `94b62eb` to `07b46fd` [00:34:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#8318: Bump extensions/Echo from `bc2a368` to `c713726` - 13https://github.com/miraheze/mediawiki/pull/8318 [00:34:34] [url] Page not found · GitHub · GitHub | github.com [00:34:35] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/RevisionSlider-07b46fd - 13https://github.com/miraheze/mediawiki [00:34:35] [url] Page not found · GitHub · GitHub | github.com [00:34:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/StructuredNavigation-4171d8b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b7da5ed1062b [00:34:38] [02miraheze/mediawiki] 07dependabot[bot] 03b7da5ed - Bump extensions/StructuredNavigation from `310e3ef` to `4171d8b` [00:34:40] [02mediawiki] 07dependabot[bot] opened pull request 03#8329: Bump extensions/StructuredNavigation from `310e3ef` to `4171d8b` - 13https://github.com/miraheze/mediawiki/pull/8329 [00:34:40] ... [00:34:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/StructuredNavigation-4171d8b - 13https://github.com/miraheze/mediawiki [00:34:41] [url] Page not found · GitHub · GitHub | github.com [00:34:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#8320: Bump extensions/WikibaseQualityConstraints from `862650e` to `852f7a9` - 13https://github.com/miraheze/mediawiki/pull/8320 [00:34:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MultimediaViewer-4b670a6 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/821a0cdaa132 [00:34:46] [02miraheze/mediawiki] 07dependabot[bot] 03821a0cd - Bump extensions/MultimediaViewer from `38aa5be` to `4b670a6` [00:34:47] [02mediawiki] 07dependabot[bot] opened pull request 03#8330: Bump extensions/MultimediaViewer from `38aa5be` to `4b670a6` - 13https://github.com/miraheze/mediawiki/pull/8330 [00:34:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#8317: Bump extensions/Translate from `d7618b5` to `ffbe9ee` - 13https://github.com/miraheze/mediawiki/pull/8317 [00:34:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MultimediaViewer-4b670a6 - 13https://github.com/miraheze/mediawiki [00:34:52] [02mediawiki] 07github-actions[bot] labeled pull request 03#8322: Bump extensions/UploadsLink from `bb40cef` to `bc21447` - 13https://github.com/miraheze/mediawiki/pull/8322 [00:34:53] [02mediawiki] 07github-actions[bot] labeled pull request 03#8317: Bump extensions/Translate from `d7618b5` to `ffbe9ee` - 13https://github.com/miraheze/mediawiki/pull/8317 [00:34:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#8321: Bump extensions/ApprovedRevs from `d4b29d5` to `2c74834` - 13https://github.com/miraheze/mediawiki/pull/8321 [00:34:56] [02mediawiki] 07github-actions[bot] labeled pull request 03#8317: Bump extensions/Translate from `d7618b5` to `ffbe9ee` - 13https://github.com/miraheze/mediawiki/pull/8317 [00:34:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#8321: Bump extensions/ApprovedRevs from `d4b29d5` to `2c74834` - 13https://github.com/miraheze/mediawiki/pull/8321 [00:34:59] [02mediawiki] 07github-actions[bot] labeled pull request 03#8317: Bump extensions/Translate from `d7618b5` to `ffbe9ee` - 13https://github.com/miraheze/mediawiki/pull/8317 [00:35:01] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Vector-14c3326 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5bc578027d67 [00:35:02] [02miraheze/mediawiki] 07dependabot[bot] 035bc5780 - Bump skins/Vector from `412d327` to `14c3326` [00:35:04] [02mediawiki] 07dependabot[bot] opened pull request 03#8331: Bump skins/Vector from `412d327` to `14c3326` - 13https://github.com/miraheze/mediawiki/pull/8331 [00:35:05] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Vector-14c3326 - 13https://github.com/miraheze/mediawiki [00:35:07] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/TemplateWizard-ca75976 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/291bcb8f6e92 [00:35:08] [02miraheze/mediawiki] 07dependabot[bot] 03291bcb8 - Bump extensions/TemplateWizard from `1edb96d` to `ca75976` [00:35:10] [02mediawiki] 07dependabot[bot] opened pull request 03#8332: Bump extensions/TemplateWizard from `1edb96d` to `ca75976` - 13https://github.com/miraheze/mediawiki/pull/8332 [00:35:11] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/TemplateWizard-ca75976 - 13https://github.com/miraheze/mediawiki [00:35:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Video-93356d1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0de8759d7495 [00:35:14] [02miraheze/mediawiki] 07dependabot[bot] 030de8759 - Bump extensions/Video from `aadd04f` to `93356d1` [00:35:16] [02mediawiki] 07dependabot[bot] opened pull request 03#8333: Bump extensions/Video from `aadd04f` to `93356d1` - 13https://github.com/miraheze/mediawiki/pull/8333 [00:35:17] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Video-93356d1 - 13https://github.com/miraheze/mediawiki [00:35:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#8323: Bump extensions/Newsletter from `8e8e2ff` to `1a9aaf9` - 13https://github.com/miraheze/mediawiki/pull/8323 [00:35:20] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MirahezeMagic-7678bd3 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f63320aa8632 [00:35:22] [02miraheze/mediawiki] 07dependabot[bot] 03f63320a - Bump extensions/MirahezeMagic from `2a7be60` to `7678bd3` [00:35:23] [02mediawiki] 07dependabot[bot] opened pull request 03#8334: Bump extensions/MirahezeMagic from `2a7be60` to `7678bd3` - 13https://github.com/miraheze/mediawiki/pull/8334 [00:35:25] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Purge-e726587 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c3ba01559f63 [00:35:26] [02miraheze/mediawiki] 07dependabot[bot] 03c3ba015 - Bump extensions/Purge from `2cae205` to `e726587` [00:35:28] [02mediawiki] 07github-actions[bot] labeled pull request 03#8324: Bump extensions/OAuth from `f7c2faf` to `0c59a55` - 13https://github.com/miraheze/mediawiki/pull/8324 [00:35:29] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MirahezeMagic-7678bd3 - 13https://github.com/miraheze/mediawiki [00:35:31] [02mediawiki] 07dependabot[bot] opened pull request 03#8335: Bump extensions/Purge from `2cae205` to `e726587` - 13https://github.com/miraheze/mediawiki/pull/8335 [00:35:32] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Purge-e726587 - 13https://github.com/miraheze/mediawiki [00:35:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#8325: Bump extensions/TimedMediaHandler from `d7e650a` to `07e93c0` - 13https://github.com/miraheze/mediawiki/pull/8325 [00:35:35] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MassMessage-75d907d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8dd13b55fb76 [00:35:37] [02miraheze/mediawiki] 07dependabot[bot] 038dd13b5 - Bump extensions/MassMessage from `73daf83` to `75d907d` [00:35:38] [02mediawiki] 07dependabot[bot] opened pull request 03#8336: Bump extensions/MassMessage from `73daf83` to `75d907d` - 13https://github.com/miraheze/mediawiki/pull/8336 [00:35:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MassMessage-75d907d - 13https://github.com/miraheze/mediawiki [00:35:41] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ConfirmEdit-f85cfef [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/721beb8230ed [00:35:43] [02miraheze/mediawiki] 07dependabot[bot] 03721beb8 - Bump extensions/ConfirmEdit from `aed1738` to `f85cfef` [00:35:44] [02mediawiki] 07github-actions[bot] labeled pull request 03#8328: Bump extensions/RevisionSlider from `94b62eb` to `07b46fd` - 13https://github.com/miraheze/mediawiki/pull/8328 [00:35:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#8329: Bump extensions/StructuredNavigation from `310e3ef` to `4171d8b` - 13https://github.com/miraheze/mediawiki/pull/8329 [00:35:47] [02mediawiki] 07dependabot[bot] opened pull request 03#8337: Bump extensions/ConfirmEdit from `aed1738` to `f85cfef` - 13https://github.com/miraheze/mediawiki/pull/8337 [00:35:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#8330: Bump extensions/MultimediaViewer from `38aa5be` to `4b670a6` - 13https://github.com/miraheze/mediawiki/pull/8330 [00:35:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ConfirmEdit-f85cfef - 13https://github.com/miraheze/mediawiki [00:35:52] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ExternalData-a5a4dc9 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5bc3b52de7fa [00:35:53] [02miraheze/mediawiki] 07dependabot[bot] 035bc3b52 - Bump extensions/ExternalData from `0be554a` to `a5a4dc9` [00:35:55] [02mediawiki] 07dependabot[bot] opened pull request 03#8338: Bump extensions/ExternalData from `0be554a` to `a5a4dc9` - 13https://github.com/miraheze/mediawiki/pull/8338 [00:35:56] [02mediawiki] 07github-actions[bot] labeled pull request 03#8326: Bump skins/Evelution from `631cf5e` to `fbe9301` - 13https://github.com/miraheze/mediawiki/pull/8326 [00:35:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ExternalData-a5a4dc9 - 13https://github.com/miraheze/mediawiki [00:35:59] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/CentralNotice-8c26ac2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/35c50c0b2fdc [00:36:01] [02miraheze/mediawiki] 07dependabot[bot] 0335c50c0 - Bump extensions/CentralNotice from `d936516` to `8c26ac2` [00:36:02] [02mediawiki] 07dependabot[bot] opened pull request 03#8339: Bump extensions/CentralNotice from `d936516` to `8c26ac2` - 13https://github.com/miraheze/mediawiki/pull/8339 [00:36:04] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/CentralNotice-8c26ac2 - 13https://github.com/miraheze/mediawiki [00:36:05] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/JsonConfig-860f758 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e31b743ee6cf [00:36:07] [02miraheze/mediawiki] 07dependabot[bot] 03e31b743 - Bump extensions/JsonConfig from `1642785` to `860f758` [00:36:08] [02mediawiki] 07dependabot[bot] opened pull request 03#8340: Bump extensions/JsonConfig from `1642785` to `860f758` - 13https://github.com/miraheze/mediawiki/pull/8340 [00:36:10] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/JsonConfig-860f758 - 13https://github.com/miraheze/mediawiki [00:36:11] [02mediawiki] 07github-actions[bot] labeled pull request 03#8327: Bump extensions/FlaggedRevs from `4bde537` to `92456b9` - 13https://github.com/miraheze/mediawiki/pull/8327 [00:36:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/SemanticMediaWiki-4fe6a36 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0d9110c9c99c [00:36:14] [02miraheze/mediawiki] 07dependabot[bot] 030d9110c - Bump extensions/SemanticMediaWiki from `da63f63` to `4fe6a36` [00:36:16] [02mediawiki] 07dependabot[bot] opened pull request 03#8341: Bump extensions/SemanticMediaWiki from `da63f63` to `4fe6a36` - 13https://github.com/miraheze/mediawiki/pull/8341 [00:36:17] [02mediawiki] 07github-actions[bot] labeled pull request 03#8327: Bump extensions/FlaggedRevs from `4bde537` to `92456b9` - 13https://github.com/miraheze/mediawiki/pull/8327 [00:36:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/SemanticMediaWiki-4fe6a36 - 13https://github.com/miraheze/mediawiki [00:36:20] [02mediawiki] 07github-actions[bot] labeled pull request 03#8327: Bump extensions/FlaggedRevs from `4bde537` to `92456b9` - 13https://github.com/miraheze/mediawiki/pull/8327 [00:36:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#8332: Bump extensions/TemplateWizard from `1edb96d` to `ca75976` - 13https://github.com/miraheze/mediawiki/pull/8332 [00:36:23] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Renameuser-69f8429 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b968e9d296cc [00:36:25] [02miraheze/mediawiki] 07dependabot[bot] 03b968e9d - Bump extensions/Renameuser from `90f80c5` to `69f8429` [00:36:26] [02mediawiki] 07github-actions[bot] labeled pull request 03#8327: Bump extensions/FlaggedRevs from `4bde537` to `92456b9` - 13https://github.com/miraheze/mediawiki/pull/8327 [00:36:28] [02mediawiki] 07dependabot[bot] opened pull request 03#8342: Bump extensions/Renameuser from `90f80c5` to `69f8429` - 13https://github.com/miraheze/mediawiki/pull/8342 [00:36:29] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Renameuser-69f8429 - 13https://github.com/miraheze/mediawiki [00:36:31] [02mediawiki] 07dependabot[bot] commented on pull request 03#8303: Bump extensions/SemanticMediaWiki from `da63f63` to `1fb586c` - 13https://github.com/miraheze/mediawiki/pull/8303#issuecomment-1200546768 [00:36:31] [url] Bump extensions/SemanticMediaWiki from `da63f63` to `1fb586c` by dependabot[bot] · Pull Request #8303 · miraheze/mediawiki · GitHub | github.com [00:36:32] [02mediawiki] 07dependabot[bot] closed pull request 03#8303: Bump extensions/SemanticMediaWiki from `da63f63` to `1fb586c` - 13https://github.com/miraheze/mediawiki/pull/8303 [00:36:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.44, 11.49, 11.16 [00:36:33] [url] Page not found · GitHub · GitHub | github.com [00:36:34] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/SemanticMediaWiki-1fb586c [00:36:35] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/SemanticMediaWiki-1fb586c - 13https://github.com/miraheze/mediawiki [00:36:35] [url] Page not found · GitHub · GitHub | github.com [00:36:37] [02mediawiki] 07dependabot[bot] commented on pull request 03#8189: Bump extensions/SemanticMediaWiki from `da63f63` to `3f964cf` - 13https://github.com/miraheze/mediawiki/pull/8189#issuecomment-1200546778 [00:36:37] [url] Bump extensions/SemanticMediaWiki from `da63f63` to `3f964cf` by dependabot[bot] · Pull Request #8189 · miraheze/mediawiki · GitHub | github.com [00:36:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/NewSignupPage-7bf4011 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/cbd0b05a69de [00:36:40] [02miraheze/mediawiki] 07dependabot[bot] 03cbd0b05 - Bump extensions/NewSignupPage from `acc48f3` to `7bf4011` [00:36:41] [02mediawiki] 07dependabot[bot] opened pull request 03#8343: Bump extensions/NewSignupPage from `acc48f3` to `7bf4011` - 13https://github.com/miraheze/mediawiki/pull/8343 [00:36:42] ... [00:36:43] [02mediawiki] 07dependabot[bot] closed pull request 03#8189: Bump extensions/SemanticMediaWiki from `da63f63` to `3f964cf` - 13https://github.com/miraheze/mediawiki/pull/8189 [00:36:43] [url] Page not found · GitHub · GitHub | github.com [00:36:44] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/NewSignupPage-7bf4011 - 13https://github.com/miraheze/mediawiki [00:36:44] [url] Page not found · GitHub · GitHub | github.com [00:36:46] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/SemanticMediaWiki-3f964cf [00:36:47] [02mediawiki] 07github-actions[bot] labeled pull request 03#8335: Bump extensions/Purge from `2cae205` to `e726587` - 13https://github.com/miraheze/mediawiki/pull/8335 [00:36:48] ... [00:36:49] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/SemanticMediaWiki-3f964cf - 13https://github.com/miraheze/mediawiki [00:36:49] [url] Page not found · GitHub · GitHub | github.com [00:36:50] [02mediawiki] 07github-actions[bot] labeled pull request 03#8334: Bump extensions/MirahezeMagic from `2a7be60` to `7678bd3` - 13https://github.com/miraheze/mediawiki/pull/8334 [00:36:51] ... [00:36:52] [02mediawiki] 07github-actions[bot] labeled pull request 03#8331: Bump skins/Vector from `412d327` to `14c3326` - 13https://github.com/miraheze/mediawiki/pull/8331 [00:36:52] [url] Page not found · GitHub · GitHub | github.com [00:36:53] [02mediawiki] 07github-actions[bot] labeled pull request 03#8334: Bump extensions/MirahezeMagic from `2a7be60` to `7678bd3` - 13https://github.com/miraheze/mediawiki/pull/8334 [00:36:53] [url] Page not found · GitHub · GitHub | github.com [00:36:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/PageSchemas-ba1ec2b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e28c84a16c53 [00:36:56] [02miraheze/mediawiki] 07dependabot[bot] 03e28c84a - Bump extensions/PageSchemas from `1d16b69` to `ba1ec2b` [00:36:58] [02mediawiki] 07dependabot[bot] opened pull request 03#8344: Bump extensions/PageSchemas from `1d16b69` to `ba1ec2b` - 13https://github.com/miraheze/mediawiki/pull/8344 [00:36:59] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/PageSchemas-ba1ec2b - 13https://github.com/miraheze/mediawiki [00:37:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#8333: Bump extensions/Video from `aadd04f` to `93356d1` - 13https://github.com/miraheze/mediawiki/pull/8333 [00:37:02] [02mediawiki] 07dependabot[bot] commented on pull request 03#8186: Bump extensions/PageSchemas from `1d16b69` to `2d20114` - 13https://github.com/miraheze/mediawiki/pull/8186#issuecomment-1200546801 [00:37:03] [url] Bump extensions/PageSchemas from `1d16b69` to `2d20114` by dependabot[bot] · Pull Request #8186 · miraheze/mediawiki · GitHub | github.com [00:37:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Graph-33bb577 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/fa7463103bca [00:37:05] [02miraheze/mediawiki] 07dependabot[bot] 03fa74631 - Bump extensions/Graph from `d180a74` to `33bb577` [00:37:07] [02mediawiki] 07dependabot[bot] opened pull request 03#8345: Bump extensions/Graph from `d180a74` to `33bb577` - 13https://github.com/miraheze/mediawiki/pull/8345 [00:37:07] ... [00:37:08] [02mediawiki] 07github-actions[bot] labeled pull request 03#8337: Bump extensions/ConfirmEdit from `aed1738` to `f85cfef` - 13https://github.com/miraheze/mediawiki/pull/8337 [00:37:09] [url] Page not found · GitHub · GitHub | github.com [00:37:10] [02mediawiki] 07dependabot[bot] closed pull request 03#8186: Bump extensions/PageSchemas from `1d16b69` to `2d20114` - 13https://github.com/miraheze/mediawiki/pull/8186 [00:37:11] [url] Page not found · GitHub · GitHub | github.com [00:37:11] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Graph-33bb577 - 13https://github.com/miraheze/mediawiki [00:37:11] ... [00:37:13] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/PageSchemas-2d20114 [00:37:14] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/PageSchemas-2d20114 - 13https://github.com/miraheze/mediawiki [00:37:14] [url] Page not found · GitHub · GitHub | github.com [00:37:16] [02mediawiki] 07github-actions[bot] labeled pull request 03#8336: Bump extensions/MassMessage from `73daf83` to `75d907d` - 13https://github.com/miraheze/mediawiki/pull/8336 [00:37:16] ... [00:37:17] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Citizen-e9139f0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/396eda4b4ccf [00:37:19] [02miraheze/mediawiki] 07dependabot[bot] 03396eda4 - Bump skins/Citizen from `e53afa5` to `e9139f0` [00:37:20] [02mediawiki] 07dependabot[bot] opened pull request 03#8346: Bump skins/Citizen from `e53afa5` to `e9139f0` - 13https://github.com/miraheze/mediawiki/pull/8346 [00:37:21] [url] Page not found · GitHub · GitHub | github.com [00:37:22] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Citizen-e9139f0 - 13https://github.com/miraheze/mediawiki [00:37:22] [url] Page not found · GitHub · GitHub | github.com [00:37:23] [02mediawiki] 07dependabot[bot] commented on pull request 03#8169: Bump skins/Citizen from `e53afa5` to `1bf51fc` - 13https://github.com/miraheze/mediawiki/pull/8169#issuecomment-1200546845 [00:37:24] [url] Bump skins/Citizen from `e53afa5` to `1bf51fc` by dependabot[bot] · Pull Request #8169 · miraheze/mediawiki · GitHub | github.com [00:37:25] [02mediawiki] 07github-actions[bot] labeled pull request 03#8339: Bump extensions/CentralNotice from `d936516` to `8c26ac2` - 13https://github.com/miraheze/mediawiki/pull/8339 [00:37:25] ... [00:37:26] [02mediawiki] 07dependabot[bot] closed pull request 03#8169: Bump skins/Citizen from `e53afa5` to `1bf51fc` - 13https://github.com/miraheze/mediawiki/pull/8169 [00:37:27] [url] Page not found · GitHub · GitHub | github.com [00:37:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MobileFrontend-104b04b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/fd5f05e7a137 [00:37:29] [02miraheze/mediawiki] 07dependabot[bot] 03fd5f05e - Bump extensions/MobileFrontend from `4e76ac5` to `104b04b` [00:37:31] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Citizen-1bf51fc [00:37:32] [02mediawiki] 07dependabot[bot] opened pull request 03#8347: Bump extensions/MobileFrontend from `4e76ac5` to `104b04b` - 13https://github.com/miraheze/mediawiki/pull/8347 [00:37:33] [url] Page not found · GitHub · GitHub | github.com [00:37:34] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Citizen-1bf51fc - 13https://github.com/miraheze/mediawiki [00:37:34] ... [00:37:35] [02mediawiki] 07dependabot[bot] commented on pull request 03#8294: Bump skins/Citizen from `e53afa5` to `3b3278c` - 13https://github.com/miraheze/mediawiki/pull/8294#issuecomment-1200546854 [00:37:36] [url] Bump skins/Citizen from `e53afa5` to `3b3278c` by dependabot[bot] · Pull Request #8294 · miraheze/mediawiki · GitHub | github.com [00:37:37] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MobileFrontend-104b04b - 13https://github.com/miraheze/mediawiki [00:37:37] [url] Page not found · GitHub · GitHub | github.com [00:37:38] [02mediawiki] 07dependabot[bot] closed pull request 03#8294: Bump skins/Citizen from `e53afa5` to `3b3278c` - 13https://github.com/miraheze/mediawiki/pull/8294 [00:37:39] [url] Page not found · GitHub · GitHub | github.com [00:37:40] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Citizen-3b3278c [00:37:41] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/skins/Citizen-3b3278c - 13https://github.com/miraheze/mediawiki [00:37:41] [url] Page not found · GitHub · GitHub | github.com [00:37:43] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/LockAuthor-44f9dfa [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1cbf888c40c4 [00:37:43] ... [00:37:44] [02miraheze/mediawiki] 07dependabot[bot] 031cbf888 - Bump extensions/LockAuthor from `1dc2678` to `44f9dfa` [00:37:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#8338: Bump extensions/ExternalData from `0be554a` to `a5a4dc9` - 13https://github.com/miraheze/mediawiki/pull/8338 [00:37:46] ... [00:37:47] [02mediawiki] 07dependabot[bot] opened pull request 03#8348: Bump extensions/LockAuthor from `1dc2678` to `44f9dfa` - 13https://github.com/miraheze/mediawiki/pull/8348 [00:37:48] [url] Page not found · GitHub · GitHub | github.com [00:37:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/LockAuthor-44f9dfa - 13https://github.com/miraheze/mediawiki [00:37:49] [url] Page not found · GitHub · GitHub | github.com [00:37:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/CommentStreams-12aedc9 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f2973c10aed2 [00:37:52] [02miraheze/mediawiki] 07dependabot[bot] 03f2973c1 - Bump extensions/CommentStreams from `6e622d4` to `12aedc9` [00:37:53] [02mediawiki] 07dependabot[bot] opened pull request 03#8349: Bump extensions/CommentStreams from `6e622d4` to `12aedc9` - 13https://github.com/miraheze/mediawiki/pull/8349 [00:37:54] ... [00:37:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/CommentStreams-12aedc9 - 13https://github.com/miraheze/mediawiki [00:37:56] [02mediawiki] 07github-actions[bot] labeled pull request 03#8340: Bump extensions/JsonConfig from `1642785` to `860f758` - 13https://github.com/miraheze/mediawiki/pull/8340 [00:37:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/CreateWiki-0a62452 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e5f71e37d86d [00:37:59] [02miraheze/mediawiki] 07dependabot[bot] 03e5f71e3 - Bump extensions/CreateWiki from `ab596c6` to `0a62452` [00:38:01] [02mediawiki] 07dependabot[bot] opened pull request 03#8350: Bump extensions/CreateWiki from `ab596c6` to `0a62452` - 13https://github.com/miraheze/mediawiki/pull/8350 [00:38:02] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/CreateWiki-0a62452 - 13https://github.com/miraheze/mediawiki [00:38:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/AJAXPoll-6d473d1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f3fbb46b5432 [00:38:05] [02miraheze/mediawiki] 07dependabot[bot] 03f3fbb46 - Bump extensions/AJAXPoll from `4d4de20` to `6d473d1` [00:38:07] [02mediawiki] 07dependabot[bot] opened pull request 03#8351: Bump extensions/AJAXPoll from `4d4de20` to `6d473d1` - 13https://github.com/miraheze/mediawiki/pull/8351 [00:38:08] [02mediawiki] 07github-actions[bot] labeled pull request 03#8344: Bump extensions/PageSchemas from `1d16b69` to `ba1ec2b` - 13https://github.com/miraheze/mediawiki/pull/8344 [00:38:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#8342: Bump extensions/Renameuser from `90f80c5` to `69f8429` - 13https://github.com/miraheze/mediawiki/pull/8342 [00:38:11] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/AJAXPoll-6d473d1 - 13https://github.com/miraheze/mediawiki [00:38:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#8344: Bump extensions/PageSchemas from `1d16b69` to `ba1ec2b` - 13https://github.com/miraheze/mediawiki/pull/8344 [00:38:14] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Disambiguator-f5f9b46 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1e9ecda48e7b [00:38:16] [02miraheze/mediawiki] 07dependabot[bot] 031e9ecda - Bump extensions/Disambiguator from `edb8790` to `f5f9b46` [00:38:17] [02mediawiki] 07dependabot[bot] opened pull request 03#8352: Bump extensions/Disambiguator from `edb8790` to `f5f9b46` - 13https://github.com/miraheze/mediawiki/pull/8352 [00:38:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Disambiguator-f5f9b46 - 13https://github.com/miraheze/mediawiki [00:38:20] [02mediawiki] 07github-actions[bot] labeled pull request 03#8345: Bump extensions/Graph from `d180a74` to `33bb577` - 13https://github.com/miraheze/mediawiki/pull/8345 [00:38:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#8343: Bump extensions/NewSignupPage from `acc48f3` to `7bf4011` - 13https://github.com/miraheze/mediawiki/pull/8343 [00:38:23] [02mediawiki] 07github-actions[bot] labeled pull request 03#8346: Bump skins/Citizen from `e53afa5` to `e9139f0` - 13https://github.com/miraheze/mediawiki/pull/8346 [00:38:25] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/WikiDiscover-9a832a7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/febe6880ad64 [00:38:26] [02miraheze/mediawiki] 07dependabot[bot] 03febe688 - Bump extensions/WikiDiscover from `d011591` to `9a832a7` [00:38:28] [02mediawiki] 07github-actions[bot] labeled pull request 03#8346: Bump skins/Citizen from `e53afa5` to `e9139f0` - 13https://github.com/miraheze/mediawiki/pull/8346 [00:38:29] [02mediawiki] 07dependabot[bot] opened pull request 03#8353: Bump extensions/WikiDiscover from `d011591` to `9a832a7` - 13https://github.com/miraheze/mediawiki/pull/8353 [00:38:31] [02mediawiki] 07github-actions[bot] labeled pull request 03#8346: Bump skins/Citizen from `e53afa5` to `e9139f0` - 13https://github.com/miraheze/mediawiki/pull/8346 [00:38:32] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/WikiDiscover-9a832a7 - 13https://github.com/miraheze/mediawiki [00:38:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.18, 10.63, 10.88 [00:38:34] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/PageTriage-c1ba244 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/508324792100 [00:38:35] [02miraheze/mediawiki] 07dependabot[bot] 035083247 - Bump extensions/PageTriage from `5abf2cc` to `c1ba244` [00:38:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#8341: Bump extensions/SemanticMediaWiki from `da63f63` to `4fe6a36` - 13https://github.com/miraheze/mediawiki/pull/8341 [00:38:38] [02mediawiki] 07dependabot[bot] opened pull request 03#8354: Bump extensions/PageTriage from `5abf2cc` to `c1ba244` - 13https://github.com/miraheze/mediawiki/pull/8354 [00:38:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#8341: Bump extensions/SemanticMediaWiki from `da63f63` to `4fe6a36` - 13https://github.com/miraheze/mediawiki/pull/8341 [00:38:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/PageTriage-c1ba244 - 13https://github.com/miraheze/mediawiki [00:38:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#8341: Bump extensions/SemanticMediaWiki from `da63f63` to `4fe6a36` - 13https://github.com/miraheze/mediawiki/pull/8341 [00:38:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/DiscussionTools-c06f0fb [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8559d2f79217 [00:38:46] [02miraheze/mediawiki] 07dependabot[bot] 038559d2f - Bump extensions/DiscussionTools from `34706ee` to `c06f0fb` [00:38:47] [02mediawiki] 07dependabot[bot] opened pull request 03#8355: Bump extensions/DiscussionTools from `34706ee` to `c06f0fb` - 13https://github.com/miraheze/mediawiki/pull/8355 [00:38:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#8348: Bump extensions/LockAuthor from `1dc2678` to `44f9dfa` - 13https://github.com/miraheze/mediawiki/pull/8348 [00:38:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/DiscussionTools-c06f0fb - 13https://github.com/miraheze/mediawiki [00:38:52] [02mediawiki] 07github-actions[bot] labeled pull request 03#8347: Bump extensions/MobileFrontend from `4e76ac5` to `104b04b` - 13https://github.com/miraheze/mediawiki/pull/8347 [00:38:53] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/GlobalBlocking-bccdf0c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/dc5a94b2edf1 [00:38:55] [02miraheze/mediawiki] 07dependabot[bot] 03dc5a94b - Bump extensions/GlobalBlocking from `2e933c0` to `bccdf0c` [00:38:56] [02mediawiki] 07dependabot[bot] opened pull request 03#8356: Bump extensions/GlobalBlocking from `2e933c0` to `bccdf0c` - 13https://github.com/miraheze/mediawiki/pull/8356 [00:38:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/GlobalBlocking-bccdf0c - 13https://github.com/miraheze/mediawiki [00:38:59] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/CentralAuth-ac6af3d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f48f8b3a82c3 [00:39:01] [02miraheze/mediawiki] 07dependabot[bot] 03f48f8b3 - Bump extensions/CentralAuth from `ff3859e` to `ac6af3d` [00:39:02] [02mediawiki] 07dependabot[bot] opened pull request 03#8357: Bump extensions/CentralAuth from `ff3859e` to `ac6af3d` - 13https://github.com/miraheze/mediawiki/pull/8357 [00:39:04] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/CentralAuth-ac6af3d - 13https://github.com/miraheze/mediawiki [00:39:05] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Flow-a988b0c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/452a135404f5 [00:39:07] [02miraheze/mediawiki] 07dependabot[bot] 03452a135 - Bump extensions/Flow from `8f60435` to `a988b0c` [00:39:08] [02mediawiki] 07dependabot[bot] opened pull request 03#8358: Bump extensions/Flow from `8f60435` to `a988b0c` - 13https://github.com/miraheze/mediawiki/pull/8358 [00:39:10] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Flow-a988b0c - 13https://github.com/miraheze/mediawiki [00:39:11] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/TranslationNotifications-bb7a748 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/93b336719f0e [00:39:13] [02miraheze/mediawiki] 07dependabot[bot] 0393b3367 - Bump extensions/TranslationNotifications from `9d01cf3` to `bb7a748` [00:39:14] [02mediawiki] 07dependabot[bot] opened pull request 03#8359: Bump extensions/TranslationNotifications from `9d01cf3` to `bb7a748` - 13https://github.com/miraheze/mediawiki/pull/8359 [00:39:16] [02mediawiki] 07github-actions[bot] labeled pull request 03#8349: Bump extensions/CommentStreams from `6e622d4` to `12aedc9` - 13https://github.com/miraheze/mediawiki/pull/8349 [00:39:17] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/TranslationNotifications-bb7a748 - 13https://github.com/miraheze/mediawiki [00:39:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#8350: Bump extensions/CreateWiki from `ab596c6` to `0a62452` - 13https://github.com/miraheze/mediawiki/pull/8350 [00:39:20] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Math-bc2d681 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/122895462d70 [00:39:22] [02miraheze/mediawiki] 07dependabot[bot] 031228954 - Bump extensions/Math from `08c4965` to `bc2d681` [00:39:23] [02mediawiki] 07github-actions[bot] labeled pull request 03#8350: Bump extensions/CreateWiki from `ab596c6` to `0a62452` - 13https://github.com/miraheze/mediawiki/pull/8350 [00:39:25] [02mediawiki] 07dependabot[bot] opened pull request 03#8360: Bump extensions/Math from `08c4965` to `bc2d681` - 13https://github.com/miraheze/mediawiki/pull/8360 [00:39:26] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Math-bc2d681 - 13https://github.com/miraheze/mediawiki [00:39:28] [02mediawiki] 07github-actions[bot] labeled pull request 03#8351: Bump extensions/AJAXPoll from `4d4de20` to `6d473d1` - 13https://github.com/miraheze/mediawiki/pull/8351 [00:39:29] [02mediawiki] 07github-actions[bot] labeled pull request 03#8353: Bump extensions/WikiDiscover from `d011591` to `9a832a7` - 13https://github.com/miraheze/mediawiki/pull/8353 [00:39:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/OATHAuth-0915f71 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8981e802dc22 [00:39:32] [02miraheze/mediawiki] 07dependabot[bot] 038981e80 - Bump extensions/OATHAuth from `7db8fba` to `0915f71` [00:39:34] [02mediawiki] 07dependabot[bot] opened pull request 03#8361: Bump extensions/OATHAuth from `7db8fba` to `0915f71` - 13https://github.com/miraheze/mediawiki/pull/8361 [00:39:35] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/OATHAuth-0915f71 - 13https://github.com/miraheze/mediawiki [00:39:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/TemplateData-4b1e166 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/792848f8112f [00:39:38] [02miraheze/mediawiki] 07dependabot[bot] 03792848f - Bump extensions/TemplateData from `bf0dadd` to `4b1e166` [00:39:40] [02mediawiki] 07dependabot[bot] opened pull request 03#8362: Bump extensions/TemplateData from `bf0dadd` to `4b1e166` - 13https://github.com/miraheze/mediawiki/pull/8362 [00:39:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/TemplateData-4b1e166 - 13https://github.com/miraheze/mediawiki [00:39:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#8352: Bump extensions/Disambiguator from `edb8790` to `f5f9b46` - 13https://github.com/miraheze/mediawiki/pull/8352 [00:39:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/WikiEditor-a56be68 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a23bfb97d58e [00:39:46] [02miraheze/mediawiki] 07dependabot[bot] 03a23bfb9 - Bump extensions/WikiEditor from `bcc7839` to `a56be68` [00:39:47] [02mediawiki] 07dependabot[bot] opened pull request 03#8363: Bump extensions/WikiEditor from `bcc7839` to `a56be68` - 13https://github.com/miraheze/mediawiki/pull/8363 [00:39:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/WikiEditor-a56be68 - 13https://github.com/miraheze/mediawiki [00:39:50] [02mediawiki] 07github-actions[bot] labeled pull request 03#8355: Bump extensions/DiscussionTools from `34706ee` to `c06f0fb` - 13https://github.com/miraheze/mediawiki/pull/8355 [00:39:52] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MultiBoilerplate-1834413 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/458fd1c3fb0b [00:39:53] [02miraheze/mediawiki] 07dependabot[bot] 03458fd1c - Bump extensions/MultiBoilerplate from `d3784c2` to `1834413` [00:39:55] [02mediawiki] 07dependabot[bot] opened pull request 03#8364: Bump extensions/MultiBoilerplate from `d3784c2` to `1834413` - 13https://github.com/miraheze/mediawiki/pull/8364 [00:39:55] [url] Page not found · GitHub · GitHub | github.com [00:39:56] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MultiBoilerplate-1834413 - 13https://github.com/miraheze/mediawiki [00:39:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#8354: Bump extensions/PageTriage from `5abf2cc` to `c1ba244` - 13https://github.com/miraheze/mediawiki/pull/8354 [00:39:59] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/TemplateStyles-9aaaa41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/de791043af0d [00:40:01] [02miraheze/mediawiki] 07dependabot[bot] 03de79104 - Bump extensions/TemplateStyles from `5face33` to `9aaaa41` [00:40:02] [02mediawiki] 07dependabot[bot] opened pull request 03#8365: Bump extensions/TemplateStyles from `5face33` to `9aaaa41` - 13https://github.com/miraheze/mediawiki/pull/8365 [00:40:04] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/TemplateStyles-9aaaa41 - 13https://github.com/miraheze/mediawiki [00:40:05] [02mediawiki] 07github-actions[bot] labeled pull request 03#8358: Bump extensions/Flow from `8f60435` to `a988b0c` - 13https://github.com/miraheze/mediawiki/pull/8358 [00:40:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#8356: Bump extensions/GlobalBlocking from `2e933c0` to `bccdf0c` - 13https://github.com/miraheze/mediawiki/pull/8356 [00:40:08] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/AntiSpoof-0462e3b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0269e3da966a [00:40:10] [02miraheze/mediawiki] 07dependabot[bot] 030269e3d - Bump extensions/AntiSpoof from `54f4eb9` to `0462e3b` [00:40:11] [02mediawiki] 07dependabot[bot] opened pull request 03#8366: Bump extensions/AntiSpoof from `54f4eb9` to `0462e3b` - 13https://github.com/miraheze/mediawiki/pull/8366 [00:40:13] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/AntiSpoof-0462e3b - 13https://github.com/miraheze/mediawiki [00:40:14] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Wikibase-c279c40 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f4cf01d821aa [00:40:16] [02miraheze/mediawiki] 07dependabot[bot] 03f4cf01d - Bump extensions/Wikibase from `b6962b2` to `c279c40` [00:40:17] [02mediawiki] 07dependabot[bot] opened pull request 03#8367: Bump extensions/Wikibase from `b6962b2` to `c279c40` - 13https://github.com/miraheze/mediawiki/pull/8367 [00:40:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Wikibase-c279c40 - 13https://github.com/miraheze/mediawiki [00:40:20] [02mediawiki] 07github-actions[bot] labeled pull request 03#8359: Bump extensions/TranslationNotifications from `9d01cf3` to `bb7a748` - 13https://github.com/miraheze/mediawiki/pull/8359 [00:40:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#8361: Bump extensions/OATHAuth from `7db8fba` to `0915f71` - 13https://github.com/miraheze/mediawiki/pull/8361 [00:40:23] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/CologneBlue-f58c976 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/3847bbda5399 [00:40:25] [02miraheze/mediawiki] 07dependabot[bot] 033847bbd - Bump skins/CologneBlue from `06d0b6a` to `f58c976` [00:40:26] [02mediawiki] 07dependabot[bot] opened pull request 03#8368: Bump skins/CologneBlue from `06d0b6a` to `f58c976` - 13https://github.com/miraheze/mediawiki/pull/8368 [00:40:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/CologneBlue-f58c976 - 13https://github.com/miraheze/mediawiki [00:40:29] [02mediawiki] 07github-actions[bot] labeled pull request 03#8357: Bump extensions/CentralAuth from `ff3859e` to `ac6af3d` - 13https://github.com/miraheze/mediawiki/pull/8357 [00:40:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ProtectSite-5904a7d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/3ece48d5374c [00:40:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.08, 11.21, 11.06 [00:40:32] [02miraheze/mediawiki] 07dependabot[bot] 033ece48d - Bump extensions/ProtectSite from `72ecb3b` to `5904a7d` [00:40:34] [02mediawiki] 07dependabot[bot] opened pull request 03#8369: Bump extensions/ProtectSite from `72ecb3b` to `5904a7d` - 13https://github.com/miraheze/mediawiki/pull/8369 [00:40:35] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ProtectSite-5904a7d - 13https://github.com/miraheze/mediawiki [00:40:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#8360: Bump extensions/Math from `08c4965` to `bc2d681` - 13https://github.com/miraheze/mediawiki/pull/8360 [00:40:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/UploadWizard-043372d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d8a103e2d204 [00:40:40] [02miraheze/mediawiki] 07dependabot[bot] 03d8a103e - Bump extensions/UploadWizard from `0b1067d` to `043372d` [00:40:41] [02mediawiki] 07dependabot[bot] opened pull request 03#8370: Bump extensions/UploadWizard from `0b1067d` to `043372d` - 13https://github.com/miraheze/mediawiki/pull/8370 [00:40:43] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/UploadWizard-043372d - 13https://github.com/miraheze/mediawiki [00:40:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Babel-a518ea0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a54e1361b55b [00:40:46] [02miraheze/mediawiki] 07dependabot[bot] 03a54e136 - Bump extensions/Babel from `3f3a407` to `a518ea0` [00:40:47] [02mediawiki] 07dependabot[bot] opened pull request 03#8371: Bump extensions/Babel from `3f3a407` to `a518ea0` - 13https://github.com/miraheze/mediawiki/pull/8371 [00:40:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Babel-a518ea0 - 13https://github.com/miraheze/mediawiki [00:40:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Popups-ff2ae1c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/289c7d5e6837 [00:40:52] [02miraheze/mediawiki] 07dependabot[bot] 03289c7d5 - Bump extensions/Popups from `ce8975e` to `ff2ae1c` [00:40:53] [02mediawiki] 07dependabot[bot] opened pull request 03#8372: Bump extensions/Popups from `ce8975e` to `ff2ae1c` - 13https://github.com/miraheze/mediawiki/pull/8372 [00:40:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Popups-ff2ae1c - 13https://github.com/miraheze/mediawiki [00:40:56] [02mediawiki] 07github-actions[bot] labeled pull request 03#8364: Bump extensions/MultiBoilerplate from `d3784c2` to `1834413` - 13https://github.com/miraheze/mediawiki/pull/8364 [00:40:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ImageRating-f9792a5 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/61fc2920be05 [00:40:59] [02miraheze/mediawiki] 07dependabot[bot] 0361fc292 - Bump extensions/ImageRating from `01caa32` to `f9792a5` [00:41:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#8362: Bump extensions/TemplateData from `bf0dadd` to `4b1e166` - 13https://github.com/miraheze/mediawiki/pull/8362 [00:41:02] [02mediawiki] 07dependabot[bot] opened pull request 03#8373: Bump extensions/ImageRating from `01caa32` to `f9792a5` - 13https://github.com/miraheze/mediawiki/pull/8373 [00:41:04] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ImageRating-f9792a5 - 13https://github.com/miraheze/mediawiki [00:41:05] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/MinervaNeue-ceb7385 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c715e3c6f714 [00:41:07] [02miraheze/mediawiki] 07dependabot[bot] 03c715e3c - Bump skins/MinervaNeue from `8d9c4f3` to `ceb7385` [00:41:08] [02mediawiki] 07dependabot[bot] opened pull request 03#8374: Bump skins/MinervaNeue from `8d9c4f3` to `ceb7385` - 13https://github.com/miraheze/mediawiki/pull/8374 [00:41:10] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/MinervaNeue-ceb7385 - 13https://github.com/miraheze/mediawiki [00:41:11] [02mediawiki] 07github-actions[bot] labeled pull request 03#8363: Bump extensions/WikiEditor from `bcc7839` to `a56be68` - 13https://github.com/miraheze/mediawiki/pull/8363 [00:41:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/SocialProfile-3c6ee3c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/abf75e1982ba [00:41:14] [02miraheze/mediawiki] 07dependabot[bot] 03abf75e1 - Bump extensions/SocialProfile from `0bb3e28` to `3c6ee3c` [00:41:16] [02mediawiki] 07dependabot[bot] opened pull request 03#8375: Bump extensions/SocialProfile from `0bb3e28` to `3c6ee3c` - 13https://github.com/miraheze/mediawiki/pull/8375 [00:41:17] [02mediawiki] 07github-actions[bot] labeled pull request 03#8366: Bump extensions/AntiSpoof from `54f4eb9` to `0462e3b` - 13https://github.com/miraheze/mediawiki/pull/8366 [00:41:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/SocialProfile-3c6ee3c - 13https://github.com/miraheze/mediawiki [00:41:20] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/IncidentReporting-5fee4ab [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/18b32412f1f2 [00:41:22] [02miraheze/mediawiki] 07dependabot[bot] 0318b3241 - Bump extensions/IncidentReporting from `394dfbd` to `5fee4ab` [00:41:23] [02mediawiki] 07dependabot[bot] opened pull request 03#8376: Bump extensions/IncidentReporting from `394dfbd` to `5fee4ab` - 13https://github.com/miraheze/mediawiki/pull/8376 [00:41:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/IncidentReporting-5fee4ab - 13https://github.com/miraheze/mediawiki [00:41:26] [02mediawiki] 07github-actions[bot] labeled pull request 03#8365: Bump extensions/TemplateStyles from `5face33` to `9aaaa41` - 13https://github.com/miraheze/mediawiki/pull/8365 [00:41:28] [02mediawiki] 07github-actions[bot] labeled pull request 03#8368: Bump skins/CologneBlue from `06d0b6a` to `f58c976` - 13https://github.com/miraheze/mediawiki/pull/8368 [00:41:29] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MagicNoCache-efca4e2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7093a8b55e56 [00:41:31] [02miraheze/mediawiki] 07dependabot[bot] 037093a8b - Bump extensions/MagicNoCache from `45bfad4` to `efca4e2` [00:41:32] [02mediawiki] 07dependabot[bot] opened pull request 03#8377: Bump extensions/MagicNoCache from `45bfad4` to `efca4e2` - 13https://github.com/miraheze/mediawiki/pull/8377 [00:41:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MagicNoCache-efca4e2 - 13https://github.com/miraheze/mediawiki [00:41:35] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Nostalgia-d36b8b6 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ef4505bf3bbf [00:41:37] [02miraheze/mediawiki] 07dependabot[bot] 03ef4505b - Bump skins/Nostalgia from `19c85cd` to `d36b8b6` [00:41:38] [02mediawiki] 07dependabot[bot] opened pull request 03#8378: Bump skins/Nostalgia from `19c85cd` to `d36b8b6` - 13https://github.com/miraheze/mediawiki/pull/8378 [00:41:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Nostalgia-d36b8b6 - 13https://github.com/miraheze/mediawiki [00:41:41] [02mediawiki] 07github-actions[bot] labeled pull request 03#8369: Bump extensions/ProtectSite from `72ecb3b` to `5904a7d` - 13https://github.com/miraheze/mediawiki/pull/8369 [00:41:43] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Gadgets-2c3f6eb [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a85c030ba3d6 [00:41:44] [02miraheze/mediawiki] 07dependabot[bot] 03a85c030 - Bump extensions/Gadgets from `292247f` to `2c3f6eb` [00:41:46] [02mediawiki] 07dependabot[bot] opened pull request 03#8379: Bump extensions/Gadgets from `292247f` to `2c3f6eb` - 13https://github.com/miraheze/mediawiki/pull/8379 [00:41:47] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Gadgets-2c3f6eb - 13https://github.com/miraheze/mediawiki [00:41:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#8373: Bump extensions/ImageRating from `01caa32` to `f9792a5` - 13https://github.com/miraheze/mediawiki/pull/8373 [00:41:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ProofreadPage-6ba0a50 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/49748e3d58c6 [00:41:52] [02miraheze/mediawiki] 07dependabot[bot] 0349748e3 - Bump extensions/ProofreadPage from `e4cfc3a` to `6ba0a50` [00:41:53] [02mediawiki] 07dependabot[bot] opened pull request 03#8380: Bump extensions/ProofreadPage from `e4cfc3a` to `6ba0a50` - 13https://github.com/miraheze/mediawiki/pull/8380 [00:41:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ProofreadPage-6ba0a50 - 13https://github.com/miraheze/mediawiki [00:41:56] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/WikiLove-06bc837 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/111d99cbe75a [00:41:58] [02miraheze/mediawiki] 07dependabot[bot] 03111d99c - Bump extensions/WikiLove from `1deead4` to `06bc837` [00:41:59] [02mediawiki] 07github-actions[bot] labeled pull request 03#8372: Bump extensions/Popups from `ce8975e` to `ff2ae1c` - 13https://github.com/miraheze/mediawiki/pull/8372 [00:42:00] [url] Page not found · GitHub · GitHub | github.com [00:42:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#8371: Bump extensions/Babel from `3f3a407` to `a518ea0` - 13https://github.com/miraheze/mediawiki/pull/8371 [00:42:02] [02mediawiki] 07dependabot[bot] opened pull request 03#8381: Bump extensions/WikiLove from `1deead4` to `06bc837` - 13https://github.com/miraheze/mediawiki/pull/8381 [00:42:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#8367: Bump extensions/Wikibase from `b6962b2` to `c279c40` - 13https://github.com/miraheze/mediawiki/pull/8367 [00:42:05] [02mediawiki] 07github-actions[bot] labeled pull request 03#8374: Bump skins/MinervaNeue from `8d9c4f3` to `ceb7385` - 13https://github.com/miraheze/mediawiki/pull/8374 [00:42:07] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/WikiLove-06bc837 - 13https://github.com/miraheze/mediawiki [00:42:08] [02mediawiki] 07github-actions[bot] labeled pull request 03#8370: Bump extensions/UploadWizard from `0b1067d` to `043372d` - 13https://github.com/miraheze/mediawiki/pull/8370 [00:42:09] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [00:42:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#8375: Bump extensions/SocialProfile from `0bb3e28` to `3c6ee3c` - 13https://github.com/miraheze/mediawiki/pull/8375 [00:42:11] [02mediawiki] 07github-actions[bot] labeled pull request 03#8376: Bump extensions/IncidentReporting from `394dfbd` to `5fee4ab` - 13https://github.com/miraheze/mediawiki/pull/8376 [00:42:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Maps-ef3da4d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7885cb73e8ca [00:42:14] [02miraheze/mediawiki] 07dependabot[bot] 037885cb7 - Bump extensions/Maps from `4cd83f9` to `ef3da4d` [00:42:16] [02mediawiki] 07dependabot[bot] opened pull request 03#8382: Bump extensions/Maps from `4cd83f9` to `ef3da4d` - 13https://github.com/miraheze/mediawiki/pull/8382 [00:42:17] [02mediawiki] 07github-actions[bot] labeled pull request 03#8377: Bump extensions/MagicNoCache from `45bfad4` to `efca4e2` - 13https://github.com/miraheze/mediawiki/pull/8377 [00:42:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Maps-ef3da4d - 13https://github.com/miraheze/mediawiki [00:42:20] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/SimpleBlogPage-9f7e016 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2518bc40f437 [00:42:22] [02miraheze/mediawiki] 07dependabot[bot] 032518bc4 - Bump extensions/SimpleBlogPage from `3173559` to `9f7e016` [00:42:23] [02mediawiki] 07dependabot[bot] commented on pull request 03#8130: Bump extensions/Maps from `4cd83f9` to `b7798a0` - 13https://github.com/miraheze/mediawiki/pull/8130#issuecomment-1200547566 [00:42:24] [url] Bump extensions/Maps from `4cd83f9` to `b7798a0` by dependabot[bot] · Pull Request #8130 · miraheze/mediawiki · GitHub | github.com [00:42:25] [02mediawiki] 07dependabot[bot] opened pull request 03#8383: Bump extensions/SimpleBlogPage from `3173559` to `9f7e016` - 13https://github.com/miraheze/mediawiki/pull/8383 [00:42:25] [url] Page not found · GitHub · GitHub | github.com [00:42:26] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/SimpleBlogPage-9f7e016 - 13https://github.com/miraheze/mediawiki [00:42:27] ... [00:42:28] [02mediawiki] 07dependabot[bot] closed pull request 03#8130: Bump extensions/Maps from `4cd83f9` to `b7798a0` - 13https://github.com/miraheze/mediawiki/pull/8130 [00:42:28] ... [00:42:29] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Maps-b7798a0 [00:42:31] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Maps-b7798a0 - 13https://github.com/miraheze/mediawiki [00:42:31] [url] Page not found · GitHub · GitHub | github.com [00:42:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.41, 11.31, 11.12 [00:42:32] [02mediawiki] 07dependabot[bot] commented on pull request 03#8290: Bump extensions/Maps from `4cd83f9` to `cf590df` - 13https://github.com/miraheze/mediawiki/pull/8290#issuecomment-1200547578 [00:42:33] [url] Bump extensions/Maps from `4cd83f9` to `cf590df` by dependabot[bot] · Pull Request #8290 · miraheze/mediawiki · GitHub | github.com [00:42:34] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/GrowthExperiments-04a1453 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/29b07d749d59 [00:42:35] [02miraheze/mediawiki] 07dependabot[bot] 0329b07d7 - Bump extensions/GrowthExperiments from `8d07fab` to `04a1453` [00:42:37] [02mediawiki] 07dependabot[bot] closed pull request 03#8290: Bump extensions/Maps from `4cd83f9` to `cf590df` - 13https://github.com/miraheze/mediawiki/pull/8290 [00:42:37] [url] Page not found · GitHub · GitHub | github.com [00:42:38] [02mediawiki] 07dependabot[bot] opened pull request 03#8384: Bump extensions/GrowthExperiments from `8d07fab` to `04a1453` - 13https://github.com/miraheze/mediawiki/pull/8384 [00:42:39] [url] Page not found · GitHub · GitHub | github.com [00:42:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/GrowthExperiments-04a1453 - 13https://github.com/miraheze/mediawiki [00:42:40] [url] Page not found · GitHub · GitHub | github.com [00:42:41] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Maps-cf590df [00:42:43] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/Maps-cf590df - 13https://github.com/miraheze/mediawiki [00:42:43] ... [00:42:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MintyDocs-591a2f2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1c2ec2e0dd8e [00:42:46] [02miraheze/mediawiki] 07dependabot[bot] 031c2ec2e - Bump extensions/MintyDocs from `b39484a` to `591a2f2` [00:42:47] [02mediawiki] 07dependabot[bot] opened pull request 03#8385: Bump extensions/MintyDocs from `b39484a` to `591a2f2` - 13https://github.com/miraheze/mediawiki/pull/8385 [00:42:48] [url] Page not found · GitHub · GitHub | github.com [00:42:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MintyDocs-591a2f2 - 13https://github.com/miraheze/mediawiki [00:42:49] ... [00:42:50] [02mediawiki] 07dependabot[bot] commented on pull request 03#8182: Bump extensions/MintyDocs from `b39484a` to `3b8c94a` - 13https://github.com/miraheze/mediawiki/pull/8182#issuecomment-1200547625 [00:42:51] [url] Bump extensions/MintyDocs from `b39484a` to `3b8c94a` by dependabot[bot] · Pull Request #8182 · miraheze/mediawiki · GitHub | github.com [00:42:52] [02mediawiki] 07dependabot[bot] closed pull request 03#8182: Bump extensions/MintyDocs from `b39484a` to `3b8c94a` - 13https://github.com/miraheze/mediawiki/pull/8182 [00:42:52] [url] Page not found · GitHub · GitHub | github.com [00:42:53] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/VisualEditor-27af982 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/125d22adffe9 [00:42:55] [02miraheze/mediawiki] 07dependabot[bot] 03125d22a - Bump extensions/VisualEditor from `488fc43` to `27af982` [00:42:56] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/MintyDocs-3b8c94a [00:42:58] [02mediawiki] 07dependabot[bot] opened pull request 03#8386: Bump extensions/VisualEditor from `488fc43` to `27af982` - 13https://github.com/miraheze/mediawiki/pull/8386 [00:42:58] ... [00:42:59] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/MintyDocs-3b8c94a - 13https://github.com/miraheze/mediawiki [00:43:00] [url] Page not found · GitHub · GitHub | github.com [00:43:01] [02mediawiki] 07dependabot[bot] commented on pull request 03#8296: Bump extensions/MintyDocs from `b39484a` to `c6e821b` - 13https://github.com/miraheze/mediawiki/pull/8296#issuecomment-1200547639 [00:43:01] [url] Bump extensions/MintyDocs from `b39484a` to `c6e821b` by dependabot[bot] · Pull Request #8296 · miraheze/mediawiki · GitHub | github.com [00:43:02] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/VisualEditor-27af982 - 13https://github.com/miraheze/mediawiki [00:43:03] [url] Page not found · GitHub · GitHub | github.com [00:43:04] [02mediawiki] 07dependabot[bot] closed pull request 03#8296: Bump extensions/MintyDocs from `b39484a` to `c6e821b` - 13https://github.com/miraheze/mediawiki/pull/8296 [00:43:04] [url] Page not found · GitHub · GitHub | github.com [00:43:05] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/MintyDocs-c6e821b [00:43:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#8378: Bump skins/Nostalgia from `19c85cd` to `d36b8b6` - 13https://github.com/miraheze/mediawiki/pull/8378 [00:43:07] [url] Page not found · GitHub · GitHub | github.com [00:43:08] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/MintyDocs-c6e821b - 13https://github.com/miraheze/mediawiki [00:43:09] ... [00:43:10] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/UniversalLanguageSelector-06091ef [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9b66a8181b7b [00:43:11] [02miraheze/mediawiki] 07dependabot[bot] 039b66a81 - Bump extensions/UniversalLanguageSelector from `b1b48bd` to `06091ef` [00:43:13] [02mediawiki] 07dependabot[bot] opened pull request 03#8387: Bump extensions/UniversalLanguageSelector from `b1b48bd` to `06091ef` - 13https://github.com/miraheze/mediawiki/pull/8387 [00:43:13] ... [00:43:14] [02mediawiki] 07github-actions[bot] labeled pull request 03#8379: Bump extensions/Gadgets from `292247f` to `2c3f6eb` - 13https://github.com/miraheze/mediawiki/pull/8379 [00:43:15] [url] Page not found · GitHub · GitHub | github.com [00:43:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/UniversalLanguageSelector-06091ef - 13https://github.com/miraheze/mediawiki [00:43:16] ... [00:43:17] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/UrlShortener-5922244 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/fae52c95d91f [00:43:19] [02miraheze/mediawiki] 07dependabot[bot] 03fae52c9 - Bump extensions/UrlShortener from `473cad5` to `5922244` [00:43:20] [02mediawiki] 07dependabot[bot] opened pull request 03#8388: Bump extensions/UrlShortener from `473cad5` to `5922244` - 13https://github.com/miraheze/mediawiki/pull/8388 [00:43:21] [url] Page not found · GitHub · GitHub | github.com [00:43:22] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/UrlShortener-5922244 - 13https://github.com/miraheze/mediawiki [00:43:23] [02mediawiki] 07dependabot[bot] commented on pull request 03#8308: Bump extensions/UniversalLanguageSelector from `b1b48bd` to `875dac9` - 13https://github.com/miraheze/mediawiki/pull/8308#issuecomment-1200547689 [00:43:24] [url] Bump extensions/UniversalLanguageSelector from `b1b48bd` to `875dac9` by dependabot[bot] · Pull Request #8308 · miraheze/mediawiki · GitHub | github.com [00:43:25] [02mediawiki] 07github-actions[bot] labeled pull request 03#8381: Bump extensions/WikiLove from `1deead4` to `06bc837` - 13https://github.com/miraheze/mediawiki/pull/8381 [00:43:25] [url] Page not found · GitHub · GitHub | github.com [00:43:26] [02mediawiki] 07github-actions[bot] labeled pull request 03#8380: Bump extensions/ProofreadPage from `e4cfc3a` to `6ba0a50` - 13https://github.com/miraheze/mediawiki/pull/8380 [00:43:27] [url] Page not found · GitHub · GitHub | github.com [00:43:28] [02mediawiki] 07dependabot[bot] closed pull request 03#8308: Bump extensions/UniversalLanguageSelector from `b1b48bd` to `875dac9` - 13https://github.com/miraheze/mediawiki/pull/8308 [00:43:28] [url] Page not found · GitHub · GitHub | github.com [00:43:29] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/UniversalLanguageSelector-875dac9 - 13https://github.com/miraheze/mediawiki [00:43:30] ... [00:43:31] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/UniversalLanguageSelector-875dac9 [00:43:32] [02mediawiki] 07github-actions[bot] labeled pull request 03#8382: Bump extensions/Maps from `4cd83f9` to `ef3da4d` - 13https://github.com/miraheze/mediawiki/pull/8382 [00:43:33] ... [00:43:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#8382: Bump extensions/Maps from `4cd83f9` to `ef3da4d` - 13https://github.com/miraheze/mediawiki/pull/8382 [00:43:34] [url] Page not found · GitHub · GitHub | github.com [00:43:35] [02mediawiki] 07github-actions[bot] labeled pull request 03#8382: Bump extensions/Maps from `4cd83f9` to `ef3da4d` - 13https://github.com/miraheze/mediawiki/pull/8382 [00:43:36] ... [00:43:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Cosmos-d1163b5 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f6e1f42aa62c [00:43:38] [02miraheze/mediawiki] 07dependabot[bot] 03f6e1f42 - Bump skins/Cosmos from `4fe5e5a` to `d1163b5` [00:43:40] [02mediawiki] 07dependabot[bot] opened pull request 03#8389: Bump skins/Cosmos from `4fe5e5a` to `d1163b5` - 13https://github.com/miraheze/mediawiki/pull/8389 [00:43:40] [url] Page not found · GitHub · GitHub | github.com [00:43:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Cosmos-d1163b5 - 13https://github.com/miraheze/mediawiki [00:43:43] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Linter-4e35872 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8fc10fa50c1a [00:43:44] [02miraheze/mediawiki] 07dependabot[bot] 038fc10fa - Bump extensions/Linter from `e634ffd` to `4e35872` [00:43:46] [02mediawiki] 07dependabot[bot] opened pull request 03#8390: Bump extensions/Linter from `e634ffd` to `4e35872` - 13https://github.com/miraheze/mediawiki/pull/8390 [00:43:47] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Linter-4e35872 - 13https://github.com/miraheze/mediawiki [00:43:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#8383: Bump extensions/SimpleBlogPage from `3173559` to `9f7e016` - 13https://github.com/miraheze/mediawiki/pull/8383 [00:43:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Kartographer-db17a4d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ea7f5409974f [00:43:52] [02miraheze/mediawiki] 07dependabot[bot] 03ea7f540 - Bump extensions/Kartographer from `7451377` to `db17a4d` [00:43:53] [02mediawiki] 07dependabot[bot] opened pull request 03#8391: Bump extensions/Kartographer from `7451377` to `db17a4d` - 13https://github.com/miraheze/mediawiki/pull/8391 [00:43:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Kartographer-db17a4d - 13https://github.com/miraheze/mediawiki [00:43:56] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/CheckUser-5231d5b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b26c5495a5e2 [00:43:58] [02miraheze/mediawiki] 07dependabot[bot] 03b26c549 - Bump extensions/CheckUser from `67bd89a` to `5231d5b` [00:43:59] [02mediawiki] 07dependabot[bot] opened pull request 03#8392: Bump extensions/CheckUser from `67bd89a` to `5231d5b` - 13https://github.com/miraheze/mediawiki/pull/8392 [00:44:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#8385: Bump extensions/MintyDocs from `b39484a` to `591a2f2` - 13https://github.com/miraheze/mediawiki/pull/8385 [00:44:02] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/CheckUser-5231d5b - 13https://github.com/miraheze/mediawiki [00:44:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#8385: Bump extensions/MintyDocs from `b39484a` to `591a2f2` - 13https://github.com/miraheze/mediawiki/pull/8385 [00:44:05] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/GlobalWatchlist-0b3d5c7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a0bfefc34729 [00:44:07] [02miraheze/mediawiki] 07dependabot[bot] 03a0bfefc - Bump extensions/GlobalWatchlist from `7020ce2` to `0b3d5c7` [00:44:08] [02mediawiki] 07github-actions[bot] labeled pull request 03#8384: Bump extensions/GrowthExperiments from `8d07fab` to `04a1453` - 13https://github.com/miraheze/mediawiki/pull/8384 [00:44:10] [02mediawiki] 07dependabot[bot] opened pull request 03#8393: Bump extensions/GlobalWatchlist from `7020ce2` to `0b3d5c7` - 13https://github.com/miraheze/mediawiki/pull/8393 [00:44:11] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/GlobalWatchlist-0b3d5c7 - 13https://github.com/miraheze/mediawiki [00:44:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/GeoGebra-e9663d2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b9e3f2aea8be [00:44:14] [02miraheze/mediawiki] 07dependabot[bot] 03b9e3f2a - Bump extensions/GeoGebra from `045616b` to `e9663d2` [00:44:16] [02mediawiki] 07dependabot[bot] opened pull request 03#8394: Bump extensions/GeoGebra from `045616b` to `e9663d2` - 13https://github.com/miraheze/mediawiki/pull/8394 [00:44:17] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/GeoGebra-e9663d2 - 13https://github.com/miraheze/mediawiki [00:44:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#8387: Bump extensions/UniversalLanguageSelector from `b1b48bd` to `06091ef` - 13https://github.com/miraheze/mediawiki/pull/8387 [00:44:20] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Thanks-8041d2d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c099d50fdad0 [00:44:22] [02miraheze/mediawiki] 07dependabot[bot] 03c099d50 - Bump extensions/Thanks from `334e06d` to `8041d2d` [00:44:23] [02mediawiki] 07github-actions[bot] labeled pull request 03#8387: Bump extensions/UniversalLanguageSelector from `b1b48bd` to `06091ef` - 13https://github.com/miraheze/mediawiki/pull/8387 [00:44:25] [02mediawiki] 07dependabot[bot] opened pull request 03#8395: Bump extensions/Thanks from `334e06d` to `8041d2d` - 13https://github.com/miraheze/mediawiki/pull/8395 [00:44:26] [02mediawiki] 07github-actions[bot] labeled pull request 03#8387: Bump extensions/UniversalLanguageSelector from `b1b48bd` to `06091ef` - 13https://github.com/miraheze/mediawiki/pull/8387 [00:44:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Thanks-8041d2d - 13https://github.com/miraheze/mediawiki [00:44:29] [02mediawiki] 07github-actions[bot] labeled pull request 03#8386: Bump extensions/VisualEditor from `488fc43` to `27af982` - 13https://github.com/miraheze/mediawiki/pull/8386 [00:44:31] [02mediawiki] 07github-actions[bot] labeled pull request 03#8388: Bump extensions/UrlShortener from `473cad5` to `5922244` - 13https://github.com/miraheze/mediawiki/pull/8388 [00:44:32] [02mediawiki] 07github-actions[bot] labeled pull request 03#8389: Bump skins/Cosmos from `4fe5e5a` to `d1163b5` - 13https://github.com/miraheze/mediawiki/pull/8389 [00:44:34] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ImportDump-ab3032b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ce0df77fba81 [00:44:35] [02miraheze/mediawiki] 07dependabot[bot] 03ce0df77 - Bump extensions/ImportDump from `62f7e59` to `ab3032b` [00:44:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#8390: Bump extensions/Linter from `e634ffd` to `4e35872` - 13https://github.com/miraheze/mediawiki/pull/8390 [00:44:38] [02mediawiki] 07dependabot[bot] opened pull request 03#8396: Bump extensions/ImportDump from `62f7e59` to `ab3032b` - 13https://github.com/miraheze/mediawiki/pull/8396 [00:44:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#8389: Bump skins/Cosmos from `4fe5e5a` to `d1163b5` - 13https://github.com/miraheze/mediawiki/pull/8389 [00:44:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ImportDump-ab3032b - 13https://github.com/miraheze/mediawiki [00:44:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#8391: Bump extensions/Kartographer from `7451377` to `db17a4d` - 13https://github.com/miraheze/mediawiki/pull/8391 [00:44:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/PdfHandler-e4bb320 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/11fcc2192200 [00:44:46] [02miraheze/mediawiki] 07dependabot[bot] 0311fcc21 - Bump extensions/PdfHandler from `52af180` to `e4bb320` [00:44:47] [02mediawiki] 07dependabot[bot] opened pull request 03#8397: Bump extensions/PdfHandler from `52af180` to `e4bb320` - 13https://github.com/miraheze/mediawiki/pull/8397 [00:44:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/PdfHandler-e4bb320 - 13https://github.com/miraheze/mediawiki [00:44:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/PropertySuggester-ddafb12 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/de5fd54f984d [00:44:52] [02miraheze/mediawiki] 07dependabot[bot] 03de5fd54 - Bump extensions/PropertySuggester from `7a42b26` to `ddafb12` [00:44:53] [02mediawiki] 07dependabot[bot] opened pull request 03#8398: Bump extensions/PropertySuggester from `7a42b26` to `ddafb12` - 13https://github.com/miraheze/mediawiki/pull/8398 [00:44:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#8393: Bump extensions/GlobalWatchlist from `7020ce2` to `0b3d5c7` - 13https://github.com/miraheze/mediawiki/pull/8393 [00:44:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/PropertySuggester-ddafb12 - 13https://github.com/miraheze/mediawiki [00:44:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/DismissableSiteNotice-8b1b481 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/dcfff29c3b30 [00:45:00] [02miraheze/mediawiki] 07dependabot[bot] 03dcfff29 - Bump extensions/DismissableSiteNotice from `48e235b` to `8b1b481` [00:45:01] [02mediawiki] 07dependabot[bot] opened pull request 03#8399: Bump extensions/DismissableSiteNotice from `48e235b` to `8b1b481` - 13https://github.com/miraheze/mediawiki/pull/8399 [00:45:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/DismissableSiteNotice-8b1b481 - 13https://github.com/miraheze/mediawiki [00:45:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/BetaFeatures-469b7de [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d0453e9c446a [00:45:06] [02miraheze/mediawiki] 07dependabot[bot] 03d0453e9 - Bump extensions/BetaFeatures from `9e4843b` to `469b7de` [00:45:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#8394: Bump extensions/GeoGebra from `045616b` to `e9663d2` - 13https://github.com/miraheze/mediawiki/pull/8394 [00:45:09] [02mediawiki] 07dependabot[bot] opened pull request 03#8400: Bump extensions/BetaFeatures from `9e4843b` to `469b7de` - 13https://github.com/miraheze/mediawiki/pull/8400 [00:45:10] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:45:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#8394: Bump extensions/GeoGebra from `045616b` to `e9663d2` - 13https://github.com/miraheze/mediawiki/pull/8394 [00:45:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/BetaFeatures-469b7de - 13https://github.com/miraheze/mediawiki [00:45:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#8392: Bump extensions/CheckUser from `67bd89a` to `5231d5b` - 13https://github.com/miraheze/mediawiki/pull/8392 [00:45:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/LinkTitles-b202254 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/49f3d90ce1bb [00:45:16] [02miraheze/mediawiki] 07dependabot[bot] 0349f3d90 - Bump extensions/LinkTitles from `c0e5396` to `b202254` [00:45:18] [02mediawiki] 07dependabot[bot] opened pull request 03#8401: Bump extensions/LinkTitles from `c0e5396` to `b202254` - 13https://github.com/miraheze/mediawiki/pull/8401 [00:45:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/LinkTitles-b202254 - 13https://github.com/miraheze/mediawiki [00:45:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/TemplateSandbox-f04f038 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/25468cc3fdd3 [00:45:22] [02miraheze/mediawiki] 07dependabot[bot] 0325468cc - Bump extensions/TemplateSandbox from `7dc7382` to `f04f038` [00:45:24] [02mediawiki] 07dependabot[bot] opened pull request 03#8402: Bump extensions/TemplateSandbox from `7dc7382` to `f04f038` - 13https://github.com/miraheze/mediawiki/pull/8402 [00:45:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/TemplateSandbox-f04f038 - 13https://github.com/miraheze/mediawiki [00:45:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/AbuseFilter-d5e8bed [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d6b8d8a12b6d [00:45:28] [02miraheze/mediawiki] 07dependabot[bot] 03d6b8d8a - Bump extensions/AbuseFilter from `b639ee1` to `d5e8bed` [00:45:30] [02mediawiki] 07dependabot[bot] opened pull request 03#8403: Bump extensions/AbuseFilter from `b639ee1` to `d5e8bed` - 13https://github.com/miraheze/mediawiki/pull/8403 [00:45:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/AbuseFilter-d5e8bed - 13https://github.com/miraheze/mediawiki [00:45:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#8395: Bump extensions/Thanks from `334e06d` to `8041d2d` - 13https://github.com/miraheze/mediawiki/pull/8395 [00:45:34] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/BlogPage-d52cc72 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/771fe999b8cc [00:45:36] [02miraheze/mediawiki] 07dependabot[bot] 03771fe99 - Bump extensions/BlogPage from `7c241eb` to `d52cc72` [00:45:37] [02mediawiki] 07dependabot[bot] opened pull request 03#8404: Bump extensions/BlogPage from `7c241eb` to `d52cc72` - 13https://github.com/miraheze/mediawiki/pull/8404 [00:45:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/BlogPage-d52cc72 - 13https://github.com/miraheze/mediawiki [00:45:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/AdminLinks-66cca57 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/db0e396de992 [00:45:42] [02miraheze/mediawiki] 07dependabot[bot] 03db0e396 - Bump extensions/AdminLinks from `025c33f` to `66cca57` [00:45:43] [02mediawiki] 07dependabot[bot] opened pull request 03#8405: Bump extensions/AdminLinks from `025c33f` to `66cca57` - 13https://github.com/miraheze/mediawiki/pull/8405 [00:45:43] [url] Page not found · GitHub · GitHub | github.com [00:45:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/AdminLinks-66cca57 - 13https://github.com/miraheze/mediawiki [00:45:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#8396: Bump extensions/ImportDump from `62f7e59` to `ab3032b` - 13https://github.com/miraheze/mediawiki/pull/8396 [00:45:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ContactPage-9913600 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/bb4c253594c4 [00:45:49] [02miraheze/mediawiki] 07dependabot[bot] 03bb4c253 - Bump extensions/ContactPage from `c3f5cea` to `9913600` [00:45:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#8400: Bump extensions/BetaFeatures from `9e4843b` to `469b7de` - 13https://github.com/miraheze/mediawiki/pull/8400 [00:45:52] [02mediawiki] 07dependabot[bot] opened pull request 03#8406: Bump extensions/ContactPage from `c3f5cea` to `9913600` - 13https://github.com/miraheze/mediawiki/pull/8406 [00:45:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#8397: Bump extensions/PdfHandler from `52af180` to `e4bb320` - 13https://github.com/miraheze/mediawiki/pull/8397 [00:45:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ContactPage-9913600 - 13https://github.com/miraheze/mediawiki [00:45:57] [02mediawiki] 07dependabot[bot] opened pull request 03#8407: Bump extensions/TitleBlacklist from `e63afbd` to `e8ed196` - 13https://github.com/miraheze/mediawiki/pull/8407 [00:45:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/TitleBlacklist-e8ed196 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a8d1ec61d412 [00:46:00] [02miraheze/mediawiki] 07dependabot[bot] 03a8d1ec6 - Bump extensions/TitleBlacklist from `e63afbd` to `e8ed196` [00:46:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#8398: Bump extensions/PropertySuggester from `7a42b26` to `ddafb12` - 13https://github.com/miraheze/mediawiki/pull/8398 [00:46:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/TitleBlacklist-e8ed196 - 13https://github.com/miraheze/mediawiki [00:46:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#8399: Bump extensions/DismissableSiteNotice from `48e235b` to `8b1b481` - 13https://github.com/miraheze/mediawiki/pull/8399 [00:46:06] [02mediawiki] 07dependabot[bot] opened pull request 03#8408: Bump extensions/PollNY from `68e62be` to `6e009de` - 13https://github.com/miraheze/mediawiki/pull/8408 [00:46:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#8399: Bump extensions/DismissableSiteNotice from `48e235b` to `8b1b481` - 13https://github.com/miraheze/mediawiki/pull/8399 [00:46:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/PollNY-6e009de [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/080771995de3 [00:46:10] [02miraheze/mediawiki] 07dependabot[bot] 030807719 - Bump extensions/PollNY from `68e62be` to `6e009de` [00:46:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/PollNY-6e009de - 13https://github.com/miraheze/mediawiki [00:46:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/QuizGame-a935e71 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/311fb054c682 [00:46:15] [02miraheze/mediawiki] 07dependabot[bot] 03311fb05 - Bump extensions/QuizGame from `24b3daa` to `a935e71` [00:46:16] [02mediawiki] 07dependabot[bot] opened pull request 03#8409: Bump extensions/QuizGame from `24b3daa` to `a935e71` - 13https://github.com/miraheze/mediawiki/pull/8409 [00:46:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/QuizGame-a935e71 - 13https://github.com/miraheze/mediawiki [00:46:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#8401: Bump extensions/LinkTitles from `c0e5396` to `b202254` - 13https://github.com/miraheze/mediawiki/pull/8401 [00:46:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Cite-374d12b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d48c2d396b38 [00:46:22] [02miraheze/mediawiki] 07dependabot[bot] 03d48c2d3 - Bump extensions/Cite from `3e577ed` to `374d12b` [00:46:24] [02mediawiki] 07dependabot[bot] opened pull request 03#8410: Bump extensions/Cite from `3e577ed` to `374d12b` - 13https://github.com/miraheze/mediawiki/pull/8410 [00:46:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Cite-374d12b - 13https://github.com/miraheze/mediawiki [00:46:27] [02mediawiki] 07github-actions[bot] labeled pull request 03#8402: Bump extensions/TemplateSandbox from `7dc7382` to `f04f038` - 13https://github.com/miraheze/mediawiki/pull/8402 [00:46:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/LinkSuggest-292e501 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8ce279dd815b [00:46:30] [02miraheze/mediawiki] 07dependabot[bot] 038ce279d - Bump extensions/LinkSuggest from `635fc63` to `292e501` [00:46:31] [02mediawiki] 07dependabot[bot] opened pull request 03#8411: Bump extensions/LinkSuggest from `635fc63` to `292e501` - 13https://github.com/miraheze/mediawiki/pull/8411 [00:46:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/LinkSuggest-292e501 - 13https://github.com/miraheze/mediawiki [00:46:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#8404: Bump extensions/BlogPage from `7c241eb` to `d52cc72` - 13https://github.com/miraheze/mediawiki/pull/8404 [00:46:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Interwiki-2f2529f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/de7562b7223e [00:46:37] [02miraheze/mediawiki] 07dependabot[bot] 03de7562b - Bump extensions/Interwiki from `3e240f9` to `2f2529f` [00:46:39] [02mediawiki] 07dependabot[bot] opened pull request 03#8412: Bump extensions/Interwiki from `3e240f9` to `2f2529f` - 13https://github.com/miraheze/mediawiki/pull/8412 [00:46:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Interwiki-2f2529f - 13https://github.com/miraheze/mediawiki [00:46:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#8408: Bump extensions/PollNY from `68e62be` to `6e009de` - 13https://github.com/miraheze/mediawiki/pull/8408 [00:46:43] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Cargo-ee86e5f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/af48d0c0f52d [00:46:45] [02miraheze/mediawiki] 07dependabot[bot] 03af48d0c - Bump extensions/Cargo from `5ecfef7` to `ee86e5f` [00:46:46] [02mediawiki] 07dependabot[bot] opened pull request 03#8413: Bump extensions/Cargo from `5ecfef7` to `ee86e5f` - 13https://github.com/miraheze/mediawiki/pull/8413 [00:46:48] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Cargo-ee86e5f - 13https://github.com/miraheze/mediawiki [00:46:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#8406: Bump extensions/ContactPage from `c3f5cea` to `9913600` - 13https://github.com/miraheze/mediawiki/pull/8406 [00:46:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#8403: Bump extensions/AbuseFilter from `b639ee1` to `d5e8bed` - 13https://github.com/miraheze/mediawiki/pull/8403 [00:46:52] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/PageForms-da3886e [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/82088a0c2911 [00:46:54] [02miraheze/mediawiki] 07dependabot[bot] 0382088a0 - Bump extensions/PageForms from `7730130` to `da3886e` [00:46:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#8405: Bump extensions/AdminLinks from `025c33f` to `66cca57` - 13https://github.com/miraheze/mediawiki/pull/8405 [00:46:57] [02mediawiki] 07dependabot[bot] opened pull request 03#8414: Bump extensions/PageForms from `7730130` to `da3886e` - 13https://github.com/miraheze/mediawiki/pull/8414 [00:46:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/PageForms-da3886e - 13https://github.com/miraheze/mediawiki [00:47:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#8409: Bump extensions/QuizGame from `24b3daa` to `a935e71` - 13https://github.com/miraheze/mediawiki/pull/8409 [00:47:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#8407: Bump extensions/TitleBlacklist from `e63afbd` to `e8ed196` - 13https://github.com/miraheze/mediawiki/pull/8407 [00:47:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/IPInfo-8655a1f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7ceb28dd55ac [00:47:04] [02miraheze/mediawiki] 07dependabot[bot] 037ceb28d - Bump extensions/IPInfo from `a1795f0` to `8655a1f` [00:47:06] [02mediawiki] 07dependabot[bot] commented on pull request 03#8295: Bump extensions/PageForms from `7730130` to `9d6e328` - 13https://github.com/miraheze/mediawiki/pull/8295#issuecomment-1200548323 [00:47:06] [url] Bump extensions/PageForms from `7730130` to `9d6e328` by dependabot[bot] · Pull Request #8295 · miraheze/mediawiki · GitHub | github.com [00:47:07] [02mediawiki] 07dependabot[bot] opened pull request 03#8415: Bump extensions/IPInfo from `a1795f0` to `8655a1f` - 13https://github.com/miraheze/mediawiki/pull/8415 [00:47:08] [url] Page not found · GitHub · GitHub | github.com [00:47:09] [02mediawiki] 07dependabot[bot] closed pull request 03#8295: Bump extensions/PageForms from `7730130` to `9d6e328` - 13https://github.com/miraheze/mediawiki/pull/8295 [00:47:09] [url] Page not found · GitHub · GitHub | github.com [00:47:10] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/PageForms-9d6e328 [00:47:12] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/PageForms-9d6e328 - 13https://github.com/miraheze/mediawiki [00:47:12] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 2.809 second response time [00:47:12] ... [00:47:13] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/IPInfo-8655a1f - 13https://github.com/miraheze/mediawiki [00:47:13] ... [00:47:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#8410: Bump extensions/Cite from `3e577ed` to `374d12b` - 13https://github.com/miraheze/mediawiki/pull/8410 [00:47:15] [url] Page not found · GitHub · GitHub | github.com [00:47:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/WikibaseLexeme-efbfc23 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1347eee2b47f [00:47:18] [02miraheze/mediawiki] 07dependabot[bot] 031347eee - Bump extensions/WikibaseLexeme from `fed9760` to `efbfc23` [00:47:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.12, 9.80, 9.45 [00:47:19] [02mediawiki] 07dependabot[bot] opened pull request 03#8416: Bump extensions/WikibaseLexeme from `fed9760` to `efbfc23` - 13https://github.com/miraheze/mediawiki/pull/8416 [00:47:20] ... [00:47:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/WikibaseLexeme-efbfc23 - 13https://github.com/miraheze/mediawiki [00:47:21] [url] Page not found · GitHub · GitHub | github.com [00:47:22] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/Comments-6d86851 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a26b19440d29 [00:47:24] [02miraheze/mediawiki] 07dependabot[bot] 03a26b194 - Bump extensions/Comments from `3dfebef` to `6d86851` [00:47:25] [02mediawiki] 07dependabot[bot] opened pull request 03#8417: Bump extensions/Comments from `3dfebef` to `6d86851` - 13https://github.com/miraheze/mediawiki/pull/8417 [00:47:26] [url] Page not found · GitHub · GitHub | github.com [00:47:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/Comments-6d86851 - 13https://github.com/miraheze/mediawiki [00:47:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ReplaceText-7abba10 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/783c0da38a4d [00:47:30] [02miraheze/mediawiki] 07dependabot[bot] 03783c0da - Bump extensions/ReplaceText from `0a5b408` to `7abba10` [00:47:31] [02mediawiki] 07dependabot[bot] opened pull request 03#8418: Bump extensions/ReplaceText from `0a5b408` to `7abba10` - 13https://github.com/miraheze/mediawiki/pull/8418 [00:47:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ReplaceText-7abba10 - 13https://github.com/miraheze/mediawiki [00:47:34] [02mediawiki] 07dependabot[bot] opened pull request 03#8419: Bump extensions/WikiForum from `a5424fe` to `702ab25` - 13https://github.com/miraheze/mediawiki/pull/8419 [00:47:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/WikiForum-702ab25 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8edfa4d5c0b2 [00:47:37] [02miraheze/mediawiki] 07dependabot[bot] 038edfa4d - Bump extensions/WikiForum from `a5424fe` to `702ab25` [00:47:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#8411: Bump extensions/LinkSuggest from `635fc63` to `292e501` - 13https://github.com/miraheze/mediawiki/pull/8411 [00:47:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/WikiForum-702ab25 - 13https://github.com/miraheze/mediawiki [00:47:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#8412: Bump extensions/Interwiki from `3e240f9` to `2f2529f` - 13https://github.com/miraheze/mediawiki/pull/8412 [00:47:43] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/StandardDialogs-1af2fe7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d9640ade5adc [00:47:45] [02miraheze/mediawiki] 07dependabot[bot] 03d9640ad - Bump extensions/StandardDialogs from `ac900d9` to `1af2fe7` [00:47:46] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/StandardDialogs-1af2fe7 - 13https://github.com/miraheze/mediawiki [00:47:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#8415: Bump extensions/IPInfo from `a1795f0` to `8655a1f` - 13https://github.com/miraheze/mediawiki/pull/8415 [00:47:49] [02mediawiki] 07dependabot[bot] opened pull request 03#8420: Bump extensions/StandardDialogs from `ac900d9` to `1af2fe7` - 13https://github.com/miraheze/mediawiki/pull/8420 [00:47:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#8413: Bump extensions/Cargo from `5ecfef7` to `ee86e5f` - 13https://github.com/miraheze/mediawiki/pull/8413 [00:47:52] [02mediawiki] 07github-actions[bot] labeled pull request 03#8413: Bump extensions/Cargo from `5ecfef7` to `ee86e5f` - 13https://github.com/miraheze/mediawiki/pull/8413 [00:47:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Mirage-d812d88 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8e7017b1ae88 [00:47:55] [02miraheze/mediawiki] 07dependabot[bot] 038e7017b - Bump skins/Mirage from `20de5dd` to `d812d88` [00:47:57] [02mediawiki] 07dependabot[bot] opened pull request 03#8421: Bump skins/Mirage from `20de5dd` to `d812d88` - 13https://github.com/miraheze/mediawiki/pull/8421 [00:47:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Mirage-d812d88 - 13https://github.com/miraheze/mediawiki [00:48:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/InputBox-61fe8e3 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/864408b65dd9 [00:48:01] [02miraheze/mediawiki] 07dependabot[bot] 03864408b - Bump extensions/InputBox from `2795cf1` to `61fe8e3` [00:48:03] [02mediawiki] 07dependabot[bot] opened pull request 03#8422: Bump extensions/InputBox from `2795cf1` to `61fe8e3` - 13https://github.com/miraheze/mediawiki/pull/8422 [00:48:04] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/InputBox-61fe8e3 - 13https://github.com/miraheze/mediawiki [00:48:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/cldr-96fdbcf [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/30f832843530 [00:48:07] [02miraheze/mediawiki] 07dependabot[bot] 0330f8328 - Bump extensions/cldr from `6f74f83` to `96fdbcf` [00:48:09] [02mediawiki] 07dependabot[bot] opened pull request 03#8423: Bump extensions/cldr from `6f74f83` to `96fdbcf` - 13https://github.com/miraheze/mediawiki/pull/8423 [00:48:10] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/cldr-96fdbcf - 13https://github.com/miraheze/mediawiki [00:48:12] [02mediawiki] 07github-actions[bot] labeled pull request 03#8414: Bump extensions/PageForms from `7730130` to `da3886e` - 13https://github.com/miraheze/mediawiki/pull/8414 [00:48:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/LoginNotify-88b6355 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/65a7bdabbfef [00:48:14] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.28, 1.81, 1.66 [00:48:15] [02miraheze/mediawiki] 07dependabot[bot] 0365a7bda - Bump extensions/LoginNotify from `68a4556` to `88b6355` [00:48:16] [02mediawiki] 07dependabot[bot] opened pull request 03#8424: Bump extensions/LoginNotify from `68a4556` to `88b6355` - 13https://github.com/miraheze/mediawiki/pull/8424 [00:48:18] [02mediawiki] 07github-actions[bot] labeled pull request 03#8414: Bump extensions/PageForms from `7730130` to `da3886e` - 13https://github.com/miraheze/mediawiki/pull/8414 [00:48:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/LoginNotify-88b6355 - 13https://github.com/miraheze/mediawiki [00:48:21] [02mediawiki] 07github-actions[bot] labeled pull request 03#8414: Bump extensions/PageForms from `7730130` to `da3886e` - 13https://github.com/miraheze/mediawiki/pull/8414 [00:48:22] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ParserFunctions-04bed9e [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/dd1720b384a5 [00:48:24] [02miraheze/mediawiki] 07dependabot[bot] 03dd1720b - Bump extensions/ParserFunctions from `bc6a7c6` to `04bed9e` [00:48:25] [02mediawiki] 07dependabot[bot] opened pull request 03#8425: Bump extensions/ParserFunctions from `bc6a7c6` to `04bed9e` - 13https://github.com/miraheze/mediawiki/pull/8425 [00:48:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ParserFunctions-04bed9e - 13https://github.com/miraheze/mediawiki [00:48:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/ArticleCreationWorkflow-e7e98a2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/3da26f719196 [00:48:30] [02miraheze/mediawiki] 07dependabot[bot] 033da26f7 - Bump extensions/ArticleCreationWorkflow from `70a8df2` to `e7e98a2` [00:48:31] [02mediawiki] 07github-actions[bot] labeled pull request 03#8416: Bump extensions/WikibaseLexeme from `fed9760` to `efbfc23` - 13https://github.com/miraheze/mediawiki/pull/8416 [00:48:33] [02mediawiki] 07dependabot[bot] opened pull request 03#8426: Bump extensions/ArticleCreationWorkflow from `70a8df2` to `e7e98a2` - 13https://github.com/miraheze/mediawiki/pull/8426 [00:48:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#8417: Bump extensions/Comments from `3dfebef` to `6d86851` - 13https://github.com/miraheze/mediawiki/pull/8417 [00:48:36] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/ArticleCreationWorkflow-e7e98a2 - 13https://github.com/miraheze/mediawiki [00:48:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/Mask-a3b9d67 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/704fc324e110 [00:48:39] [02miraheze/mediawiki] 07dependabot[bot] 03704fc32 - Bump skins/Mask from `c38263d` to `a3b9d67` [00:48:40] [02mediawiki] 07dependabot[bot] opened pull request 03#8427: Bump skins/Mask from `c38263d` to `a3b9d67` - 13https://github.com/miraheze/mediawiki/pull/8427 [00:48:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/Mask-a3b9d67 - 13https://github.com/miraheze/mediawiki [00:48:43] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/GlobalUsage-04323e5 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/195a05f883dc [00:48:45] [02miraheze/mediawiki] 07dependabot[bot] 03195a05f - Bump extensions/GlobalUsage from `fcc342a` to `04323e5` [00:48:46] [02mediawiki] 07dependabot[bot] opened pull request 03#8428: Bump extensions/GlobalUsage from `fcc342a` to `04323e5` - 13https://github.com/miraheze/mediawiki/pull/8428 [00:48:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#8418: Bump extensions/ReplaceText from `0a5b408` to `7abba10` - 13https://github.com/miraheze/mediawiki/pull/8418 [00:48:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/GlobalUsage-04323e5 - 13https://github.com/miraheze/mediawiki [00:48:50] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.63, 1.74, 1.51 [00:48:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/StopForumSpam-6735e6e [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2e536fc3b183 [00:48:52] [02miraheze/mediawiki] 07dependabot[bot] 032e536fc - Bump extensions/StopForumSpam from `7a45b42` to `6735e6e` [00:48:54] [02mediawiki] 07dependabot[bot] opened pull request 03#8429: Bump extensions/StopForumSpam from `7a45b42` to `6735e6e` - 13https://github.com/miraheze/mediawiki/pull/8429 [00:48:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/StopForumSpam-6735e6e - 13https://github.com/miraheze/mediawiki [00:48:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#8421: Bump skins/Mirage from `20de5dd` to `d812d88` - 13https://github.com/miraheze/mediawiki/pull/8421 [00:48:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#8420: Bump extensions/StandardDialogs from `ac900d9` to `1af2fe7` - 13https://github.com/miraheze/mediawiki/pull/8420 [00:49:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#8419: Bump extensions/WikiForum from `a5424fe` to `702ab25` - 13https://github.com/miraheze/mediawiki/pull/8419 [00:49:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#8422: Bump extensions/InputBox from `2795cf1` to `61fe8e3` - 13https://github.com/miraheze/mediawiki/pull/8422 [00:49:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#8424: Bump extensions/LoginNotify from `68a4556` to `88b6355` - 13https://github.com/miraheze/mediawiki/pull/8424 [00:49:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/DynamicPageList3-b0f6cd6 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8253abe0dda9 [00:49:06] [02miraheze/mediawiki] 07dependabot[bot] 038253abe - Bump extensions/DynamicPageList3 from `05190aa` to `b0f6cd6` [00:49:07] [02mediawiki] 07dependabot[bot] opened pull request 03#8430: Bump extensions/DynamicPageList3 from `05190aa` to `b0f6cd6` - 13https://github.com/miraheze/mediawiki/pull/8430 [00:49:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/DynamicPageList3-b0f6cd6 - 13https://github.com/miraheze/mediawiki [00:49:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#8423: Bump extensions/cldr from `6f74f83` to `96fdbcf` - 13https://github.com/miraheze/mediawiki/pull/8423 [00:49:12] [02mediawiki] 07github-actions[bot] labeled pull request 03#8425: Bump extensions/ParserFunctions from `bc6a7c6` to `04bed9e` - 13https://github.com/miraheze/mediawiki/pull/8425 [00:49:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#8426: Bump extensions/ArticleCreationWorkflow from `70a8df2` to `e7e98a2` - 13https://github.com/miraheze/mediawiki/pull/8426 [00:49:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#8427: Bump skins/Mask from `c38263d` to `a3b9d67` - 13https://github.com/miraheze/mediawiki/pull/8427 [00:49:15] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.68, 9.78, 9.49 [00:49:16] [02mediawiki] 07github-actions[bot] labeled pull request 03#8428: Bump extensions/GlobalUsage from `fcc342a` to `04323e5` - 13https://github.com/miraheze/mediawiki/pull/8428 [00:49:18] [02mediawiki] 07github-actions[bot] labeled pull request 03#8429: Bump extensions/StopForumSpam from `7a45b42` to `6735e6e` - 13https://github.com/miraheze/mediawiki/pull/8429 [00:49:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#8430: Bump extensions/DynamicPageList3 from `05190aa` to `b0f6cd6` - 13https://github.com/miraheze/mediawiki/pull/8430 [00:50:14] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.81, 1.79, 1.67 [00:50:32] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.80, 1.51, 1.26 [00:50:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.21, 11.51, 11.16 [00:50:44] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.73, 2.02, 1.63 [00:52:14] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.45, 1.70, 1.65 [00:52:32] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.60, 1.49, 1.28 [00:52:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.01, 11.36, 11.15 [00:53:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/6ff717c10296 [00:53:17] [02miraheze/mw-config] 07Universal-Omega 036ff717c - Set actor table schema migration stage to WRITE_NEW in production [00:53:19] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [00:53:20] [url] Page not found · GitHub · GitHub | github.com [00:53:20] [02mw-config] 07Universal-Omega opened pull request 03#4832: Set actor table schema migration stage to WRITE_NEW in production - 13https://github.com/miraheze/mw-config/pull/4832 [00:54:33] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.58, 1.95, 1.70 [00:54:40] miraheze/mw-config - Universal-Omega the build passed. [00:58:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.34, 1.68, 1.65 [01:02:11] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.33, 2.29, 1.90 [01:06:00] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.90, 1.71, 1.77 [01:06:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.12, 10.44, 10.49 [01:08:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.53, 9.97, 10.31 [01:09:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.60, 1.34, 1.62 [01:11:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.13, 3.21, 3.84 [01:12:32] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.50, 9.54, 10.05 [01:13:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.09, 1.78, 1.60 [01:19:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.24, 3.44, 3.66 [01:21:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.20, 3.53, 3.69 [01:23:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.38, 3.84, 3.79 [01:25:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.38, 3.76, 3.77 [01:26:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.92, 10.98, 10.45 [01:27:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.16, 3.84, 3.80 [01:32:06] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.19, 11.42, 10.76 [01:32:42] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.65, 3.29, 2.19 [01:34:02] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.92, 10.96, 10.66 [01:36:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.83, 1.55, 1.42 [01:37:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.10, 11.35, 10.87 [01:38:10] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.89, 1.32, 1.35 [01:38:32] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.78, 1.58, 1.77 [01:39:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.24, 10.70, 10.69 [01:40:32] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.38, 1.46, 1.70 [01:43:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.08, 3.87, 3.99 [01:43:43] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.52, 9.50, 10.19 [01:45:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.50, 3.94, 3.99 [01:47:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.79, 3.93 [01:49:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.04, 4.70, 4.25 [01:49:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.65, 9.80, 10.04 [01:50:33] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 11.24, 3.88, 2.20 [01:51:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.88, 10.98, 10.43 [01:55:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.51, 11.36, 10.71 [01:58:14] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.82, 1.89, 2.00 [01:59:08] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [02:02:00] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.49, 1.83, 1.97 [02:05:04] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.05, 11.41, 10.94 [02:06:14] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.07, 1.83, 1.91 [02:07:48] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.51, 1.19, 1.67 [02:07:59] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:08:14] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.59, 1.70, 1.85 [02:08:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.49, 10.63, 10.81 [02:10:41] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:55] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.105 second response time [02:16:14] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.06, 1.62, 1.73 [02:18:14] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.92, 1.70, 1.74 [02:18:42] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 7.190 second response time [02:20:14] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.57, 1.59, 1.69 [02:22:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.08, 11.30, 10.96 [02:23:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.68, 3.41, 3.87 [02:24:33] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.62, 1.92, 1.67 [02:25:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.19, 3.99, 4.02 [02:26:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.03, 11.33, 11.05 [02:28:22] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.73, 1.42, 1.53 [02:29:08] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [02:29:32] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/a7c103fc613e [02:29:33] [02miraheze/dns] 07paladox 03a7c103f - Switch elasticsearch backend to es131 [02:29:35] [02dns] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/dns [02:29:35] [url] Page not found · GitHub · GitHub | github.com [02:29:36] [02dns] 07paladox opened pull request 03#308: Switch elasticsearch backend to es131 - 13https://github.com/miraheze/dns/pull/308 [02:30:35] [02puppet] 07paladox closed pull request 03#2715: Install es131 / es141 - 13https://github.com/miraheze/puppet/pull/2715 [02:30:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±2] 13https://github.com/miraheze/puppet/compare/c07035919da6...233d632cc918 [02:30:37] [url] Comparing c07035919da6...233d632cc918 · miraheze/puppet · GitHub | github.com [02:30:38] [02miraheze/puppet] 07paladox 03233d632 - Install es131 / es141 (#2715) [02:30:40] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [02:30:41] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [02:30:42] ... [02:30:44] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.98, 2.00, 1.61 [02:32:14] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.74, 1.54, 1.56 [02:32:21] RECOVERY - es131 Puppet on es131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:32:39] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:32:40] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.69, 1.55, 1.49 [02:33:07] RECOVERY - es141 Puppet on es141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:34:05] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.18, 1.97, 1.73 [02:34:14] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.42, 1.51, 1.54 [02:34:38] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.632 second response time [02:36:00] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.53, 1.74, 1.67 [02:37:54] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.55, 1.69, 1.65 [02:38:22] PROBLEM - es131 Puppet on es131 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 1 minute ago with 2 failures. Failed resources (up to 3 shown): Service[elasticsearch],Elasticsearch_template[graylog-internal] [02:39:07] PROBLEM - es141 Puppet on es141 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 1 minute ago with 2 failures. Failed resources (up to 3 shown): Service[elasticsearch],Elasticsearch_template[graylog-internal] [02:43:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.46, 3.58, 3.96 [02:44:32] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.23, 9.37, 10.13 [02:46:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/233d632cc918...ada7e447f047 [02:46:36] [url] Comparing 233d632cc918...ada7e447f047 · miraheze/puppet · GitHub | github.com [02:46:37] [02miraheze/puppet] 07paladox 03ada7e44 - elasticsearch: downgrade to 7.10.2 [02:49:01] icinga-miraheze: Not-15ab [02:49:05] Whoops. [02:51:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.48, 3.72, 3.81 [02:51:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.07, 10.52, 10.41 [02:53:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.16, 3.48, 3.71 [02:55:07] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.36, 2.10, 1.79 [02:56:15] [02dns] 07paladox closed pull request 03#308: Switch elasticsearch backend to es131 - 13https://github.com/miraheze/dns/pull/308 [02:56:15] [url] Page not found · GitHub · GitHub | github.com [02:56:16] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/cad2bd595b3c...55e1de198484 [02:56:17] [url] Comparing cad2bd595b3c...55e1de198484 · miraheze/dns · GitHub | github.com [02:56:18] [02miraheze/dns] 07paladox 0355e1de1 - Switch elasticsearch backend to es131 (#308) [02:56:19] [02dns] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/dns [02:56:19] [url] Page not found · GitHub · GitHub | github.com [02:56:21] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-2 [02:57:02] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.71, 1.59, 1.64 [02:57:34] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.33, 1.80, 1.58 [03:02:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ada7e447f047...a7f7c76129e0 [03:02:13] [url] Comparing ada7e447f047...a7f7c76129e0 · miraheze/puppet · GitHub | github.com [03:02:14] [02miraheze/puppet] 07paladox 03a7f7c76 - elasticsearch: switch master in nginx config [03:03:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.20, 3.38, 3.52 [03:05:01] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.10, 10.82, 10.58 [03:05:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.74, 3.40, 3.51 [03:06:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.95, 11.13, 10.72 [03:13:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.24, 3.98, 3.71 [03:13:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.17, 10.42, 9.69 [03:20:32] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.39, 11.46, 10.90 [03:23:19] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.51, 9.67, 9.78 [03:24:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.40, 11.59, 11.10 [03:25:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.87, 3.79 [03:29:25] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 8.15, 5.05, 3.19 [03:31:25] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 4.78, 5.02, 3.42 [03:34:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a7f7c76129e0...f54ae2e8cb1b [03:34:46] [url] Comparing a7f7c76129e0...f54ae2e8cb1b · miraheze/puppet · GitHub | github.com [03:34:48] [02miraheze/puppet] 07paladox 03f54ae2e - Fix [03:34:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f54ae2e8cb1b...b5052437870a [03:35:00] [url] Comparing f54ae2e8cb1b...b5052437870a · miraheze/puppet · GitHub | github.com [03:35:00] [02miraheze/puppet] 07paladox 03b505243 - Fix [03:36:21] RECOVERY - es131 Puppet on es131 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [03:37:07] RECOVERY - es141 Puppet on es141 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [03:39:11] RECOVERY - es121 Puppet on es121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:39:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.08, 3.73, 3.64 [03:40:32] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.57, 8.75, 9.95 [03:41:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.28, 3.57, 3.59 [03:42:14] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.56, 1.82, 1.99 [03:44:14] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 1.93, 1.95, 2.02 [03:45:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.36, 3.77, 3.66 [03:47:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.79, 3.71, 3.65 [03:47:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/50250ec4835e [03:47:56] [02miraheze/puppet] 07paladox 0350250ec - Depool es121 [03:47:57] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [03:47:58] [url] Page not found · GitHub · GitHub | github.com [03:47:59] [02puppet] 07paladox opened pull request 03#2716: Depool es121 - 13https://github.com/miraheze/puppet/pull/2716 [03:47:59] [url] Page not found · GitHub · GitHub | github.com [03:48:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/50250ec4835e...cb2bf8d65f35 [03:48:13] [url] Comparing 50250ec4835e...cb2bf8d65f35 · miraheze/puppet · GitHub | github.com [03:48:14] [02miraheze/puppet] 07paladox 03cb2bf8d - Update es141.yaml [03:48:16] [02puppet] 07paladox synchronize pull request 03#2716: Depool es121 - 13https://github.com/miraheze/puppet/pull/2716 [03:48:16] [url] Page not found · GitHub · GitHub | github.com [03:48:29] [02puppet] 07paladox closed pull request 03#2716: Depool es121 - 13https://github.com/miraheze/puppet/pull/2716 [03:48:30] [url] Page not found · GitHub · GitHub | github.com [03:48:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/b5052437870a...3087b929e0f8 [03:48:31] [url] Comparing b5052437870a...3087b929e0f8 · miraheze/puppet · GitHub | github.com [03:48:32] [02miraheze/puppet] 07paladox 033087b92 - Depool es121 (#2716) [03:48:34] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [03:48:34] [url] Page not found · GitHub · GitHub | github.com [03:48:35] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [03:49:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.86, 3.87, 3.69 [03:50:14] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.60, 1.92, 2.00 [03:51:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.14, 3.59, 3.62 [03:57:02] PROBLEM - es101 Puppet on es101 is CRITICAL: connect to address 2a10:6740::6:103 port 5666: No route to hostconnect to host 2a10:6740::6:103 port 5666: No route to host [03:57:16] !log shutdown es101/111/121 [03:57:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.94, 4.14, 3.75 [03:57:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:58:53] PROBLEM - es111 Puppet on es111 is CRITICAL: connect to address 2a10:6740::6:202 port 5666: No route to hostconnect to host 2a10:6740::6:202 port 5666: No route to host [03:58:55] PROBLEM - es101 PowerDNS Recursor on es101 is CRITICAL: connect to address 2a10:6740::6:103 port 5666: No route to hostconnect to host 2a10:6740::6:103 port 5666: No route to host [03:59:02] PROBLEM - Host es101 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:103) [03:59:03] PROBLEM - es111 Current Load on es111 is CRITICAL: connect to address 2a10:6740::6:202 port 5666: No route to hostconnect to host 2a10:6740::6:202 port 5666: No route to host [03:59:09] PROBLEM - ping6 on es111 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:202) [03:59:23] PROBLEM - es111 SSH on es111 is CRITICAL: connect to address 2a10:6740::6:202 and port 22: No route to host [03:59:52] PROBLEM - es121 Current Load on es121 is CRITICAL: connect to address 2a10:6740::6:303 port 5666: No route to hostconnect to host 2a10:6740::6:303 port 5666: No route to host [03:59:54] PROBLEM - Host es111 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:202) [03:59:56] PROBLEM - es121 NTP time on es121 is CRITICAL: connect to address 2a10:6740::6:303 port 5666: No route to hostconnect to host 2a10:6740::6:303 port 5666: No route to host [04:00:03] PROBLEM - es121 Disk Space on es121 is CRITICAL: connect to address 2a10:6740::6:303 port 5666: No route to hostconnect to host 2a10:6740::6:303 port 5666: No route to host [04:00:03] PROBLEM - ping6 on es121 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:303) [04:00:14] PROBLEM - es121 Puppet on es121 is CRITICAL: connect to address 2a10:6740::6:303 port 5666: No route to hostconnect to host 2a10:6740::6:303 port 5666: No route to host [04:00:14] PROBLEM - es121 ferm_active on es121 is CRITICAL: connect to address 2a10:6740::6:303 port 5666: No route to hostconnect to host 2a10:6740::6:303 port 5666: No route to host [04:00:14] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.09, 1.34, 1.67 [04:00:28] PROBLEM - es121 SSH on es121 is CRITICAL: connect to address 2a10:6740::6:303 and port 22: No route to host [04:00:54] PROBLEM - Host es121 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:303) [04:04:32] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.26, 2.03, 1.66 [04:06:22] PROBLEM - es131 Puppet on es131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 26 seconds ago with 1 failures. Failed resources (up to 3 shown): Elasticsearch_template[graylog-internal] [04:06:32] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.10, 1.59, 1.54 [04:07:07] PROBLEM - es141 Puppet on es141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Elasticsearch_template[graylog-internal] [04:08:21] RECOVERY - es131 Puppet on es131 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [04:08:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 3.70, 3.99 [04:09:07] RECOVERY - es141 Puppet on es141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:10:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.52, 4.06, 4.09 [04:11:48] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.98, 1.66, 1.42 [04:12:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.97, 3.66, 3.94 [04:13:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.61, 3.92, 3.98 [04:13:48] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.20, 1.46, 1.38 [04:16:43] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.59, 3.31, 2.82 [04:17:48] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.98, 1.84, 1.56 [04:18:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-1/±0] 13https://github.com/miraheze/puppet/commit/44aae60fbc4a [04:18:02] [02miraheze/puppet] 07paladox 0344aae60 - Remove es101/111/121 [04:18:04] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [04:18:04] [url] Page not found · GitHub · GitHub | github.com [04:18:05] [02puppet] 07paladox opened pull request 03#2717: Remove es101/111/121 - 13https://github.com/miraheze/puppet/pull/2717 [04:18:06] ... [04:18:14] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.77, 1.68, 1.56 [04:18:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/44aae60fbc4a...4e44f0c4fc08 [04:18:23] [url] Comparing 44aae60fbc4a...4e44f0c4fc08 · miraheze/puppet · GitHub | github.com [04:18:24] [02miraheze/puppet] 07paladox 034e44f0c - Delete es121.yaml [04:18:26] [02puppet] 07paladox synchronize pull request 03#2717: Remove es101/111/121 - 13https://github.com/miraheze/puppet/pull/2717 [04:18:26] [url] Page not found · GitHub · GitHub | github.com [04:18:37] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.36, 2.98, 2.77 [04:18:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/4e44f0c4fc08...b96b808c0eee [04:18:39] [url] Comparing 4e44f0c4fc08...b96b808c0eee · miraheze/puppet · GitHub | github.com [04:18:40] [02miraheze/puppet] 07paladox 03b96b808 - Delete es111.yaml [04:18:42] [02puppet] 07paladox synchronize pull request 03#2717: Remove es101/111/121 - 13https://github.com/miraheze/puppet/pull/2717 [04:18:42] [url] Page not found · GitHub · GitHub | github.com [04:18:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.07, 3.99, 3.91 [04:18:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b96b808c0eee...5fe0ce73ec11 [04:18:58] [url] Comparing b96b808c0eee...5fe0ce73ec11 · miraheze/puppet · GitHub | github.com [04:18:59] [02miraheze/puppet] 07paladox 035fe0ce7 - Update site.pp [04:19:00] [02puppet] 07paladox synchronize pull request 03#2717: Remove es101/111/121 - 13https://github.com/miraheze/puppet/pull/2717 [04:19:00] [url] Page not found · GitHub · GitHub | github.com [04:19:09] [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [04:19:09] [url] Page not found · GitHub · GitHub | github.com [04:19:10] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11 [04:19:12] [02miraheze/puppet] 07paladox deleted branch 03The-Voidwalker-patch-3 [04:19:13] [02puppet] 07paladox deleted branch 03The-Voidwalker-patch-3 - 13https://github.com/miraheze/puppet [04:19:14] [url] Page not found · GitHub · GitHub | github.com [04:19:15] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [04:19:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.14, 4.59, 4.20 [04:19:16] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [04:19:17] ... [04:19:18] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [04:19:19] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [04:19:20] ... [04:19:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.91, 2.60, 1.89 [04:20:11] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/e940d7440e62 [04:20:12] [02miraheze/dns] 07paladox 03e940d74 - Remove es101/111/121 from dns [04:20:14] [02dns] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/dns [04:20:14] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.21, 1.55, 1.53 [04:20:14] [url] Page not found · GitHub · GitHub | github.com [04:20:15] [02dns] 07paladox opened pull request 03#309: Remove es101/111/121 from dns - 13https://github.com/miraheze/dns/pull/309 [04:20:16] ... [04:20:32] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.67, 2.02, 1.74 [04:22:32] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.46, 1.97, 1.77 [04:22:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.95, 3.93 [04:23:36] [02dns] 07paladox closed pull request 03#309: Remove es101/111/121 from dns - 13https://github.com/miraheze/dns/pull/309 [04:23:36] [url] Page not found · GitHub · GitHub | github.com [04:23:37] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/55e1de198484...f4d9c759adaa [04:23:38] [url] Comparing 55e1de198484...f4d9c759adaa · miraheze/dns · GitHub | github.com [04:23:39] [02miraheze/dns] 07paladox 03f4d9c75 - Remove es101/111/121 from dns (#309) [04:23:46] [02puppet] 07paladox closed pull request 03#2717: Remove es101/111/121 - 13https://github.com/miraheze/puppet/pull/2717 [04:23:47] [url] Page not found · GitHub · GitHub | github.com [04:23:48] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [04:23:48] [url] Page not found · GitHub · GitHub | github.com [04:23:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-3/±1] 13https://github.com/miraheze/puppet/compare/3087b929e0f8...96daf13c1c21 [04:23:50] [url] Comparing 3087b929e0f8...96daf13c1c21 · miraheze/puppet · GitHub | github.com [04:23:51] [02miraheze/puppet] 07paladox 0396daf13 - Remove es101/111/121 (#2717) [04:23:52] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [04:23:55] !log delete vms es101/111/121 [04:24:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:24:32] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.01, 1.94, 1.78 [04:26:32] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.03, 1.68, 1.70 [04:27:48] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.94, 1.77, 1.83 [04:28:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.82, 3.84 [04:31:48] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.99, 1.41, 1.67 [04:34:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.47, 9.36, 8.59 [04:36:32] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.73, 9.04, 8.56 [04:37:04] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/c42ae0e1976f [04:37:05] [02miraheze/dns] 07paladox 03c42ae0e - Add gluster102 & gluster122 to dns [04:37:07] [02dns] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/dns [04:37:07] [url] Page not found · GitHub · GitHub | github.com [04:37:08] [02dns] 07paladox opened pull request 03#310: Add gluster102 & gluster122 to dns - 13https://github.com/miraheze/dns/pull/310 [04:37:08] ... [04:37:36] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/dns/compare/c42ae0e1976f...6659508da768 [04:37:37] [url] Comparing c42ae0e1976f...6659508da768 · miraheze/dns · GitHub | github.com [04:37:38] [02miraheze/dns] 07paladox 036659508 - Update miraheze.org [04:37:39] [02dns] 07paladox synchronize pull request 03#310: Add gluster102 & gluster122 to dns - 13https://github.com/miraheze/dns/pull/310 [04:37:40] [url] Page not found · GitHub · GitHub | github.com [04:42:14] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.08, 1.84, 1.56 [04:43:59] [02dns] 07paladox closed pull request 03#310: Add gluster102 & gluster122 to dns - 13https://github.com/miraheze/dns/pull/310 [04:43:59] [url] Page not found · GitHub · GitHub | github.com [04:44:00] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/f4d9c759adaa...4873046a7804 [04:44:01] [url] Comparing f4d9c759adaa...4873046a7804 · miraheze/dns · GitHub | github.com [04:44:02] [02miraheze/dns] 07paladox 034873046 - Add gluster102 & gluster122 to dns (#310) [04:44:05] [02dns] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/dns [04:44:05] [url] Page not found · GitHub · GitHub | github.com [04:44:07] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-3 [04:48:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.16, 2.90, 3.71 [04:49:54] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [04:49:59] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [04:50:11] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:12] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:13] PROBLEM - cp31 Stunnel HTTP for mail121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [04:50:13] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:14] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [04:50:14] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.96, 1.79 [04:50:18] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [04:50:25] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:29] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:31] PROBLEM - cp30 Stunnel HTTP for test101 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [04:50:31] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [04:50:32] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [04:50:33] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:33] PROBLEM - cp30 Stunnel HTTP for mon111 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [04:50:35] PROBLEM - cp30 Stunnel HTTP for mail121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [04:50:36] PROBLEM - cp30 Stunnel HTTP for reports121 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [04:50:37] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [04:50:37] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:39] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:39] PROBLEM - mw112 HTTPS on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 0.31, 1.99, 3.28 [04:50:45] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:49] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:57] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [04:51:01] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:06] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:07] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:08] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:12] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [04:51:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 0.15, 2.13, 3.55 [04:51:16] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [04:51:19] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:20] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:20] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:25] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:25] PROBLEM - mw101 HTTPS on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:30] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:30] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [04:51:33] I guess I don't mention we're experiencing some downtime [04:51:39] *don't need to [04:51:39] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:40] PROBLEM - mw111 HTTPS on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:40] PROBLEM - mw102 HTTPS on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:47] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [04:52:01] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:52:02] PROBLEM - mw121 HTTPS on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:52:06] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:52:13] RECOVERY - cp31 Stunnel HTTP for mail121 on cp31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.252 second response time [04:52:15] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 99046 bytes in 0.795 second response time [04:52:20] PROBLEM - mw122 HTTPS on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:52:31] RECOVERY - cp30 Stunnel HTTP for test101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15954 bytes in 0.306 second response time [04:52:33] RECOVERY - cp30 Stunnel HTTP for mon111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 39169 bytes in 0.351 second response time [04:52:35] RECOVERY - cp30 Stunnel HTTP for mail121 on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 427 bytes in 0.260 second response time [04:52:36] RECOVERY - cp30 Stunnel HTTP for reports121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 10996 bytes in 0.259 second response time [04:53:16] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 0.04, 1.43, 3.12 [04:53:21] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [04:53:32] paladox: ^^^ [04:54:15] RECOVERY - mw122 HTTPS on mw122 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.011 second response time [04:54:47] PROBLEM - prometheus101 Current Load on prometheus101 is CRITICAL: CRITICAL - load average: 3.67, 1.94, 0.95 [04:54:51] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 96% [04:55:00] PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 51% [04:55:05] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [04:55:11] PROBLEM - prometheus101 APT on prometheus101 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [04:55:32] RECOVERY - mw101 HTTPS on mw101 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.010 second response time [04:55:41] RECOVERY - mw102 HTTPS on mw102 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.014 second response time [04:55:45] RECOVERY - mw111 HTTPS on mw111 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.016 second response time [04:55:49] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.070 second response time [04:56:01] RECOVERY - mw121 HTTPS on mw121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.010 second response time [04:56:03] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.330 second response time [04:56:05] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.319 second response time [04:56:07] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.319 second response time [04:56:07] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.309 second response time [04:56:11] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [04:56:14] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.332 second response time [04:56:16] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [04:56:30] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.316 second response time [04:56:31] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [04:56:37] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.313 second response time [04:56:38] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [04:56:38] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [04:56:43] PROBLEM - prometheus101 Current Load on prometheus101 is WARNING: WARNING - load average: 1.62, 1.91, 1.07 [04:56:47] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.250 second response time [04:56:47] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.316 second response time [04:56:48] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.290 second response time [04:56:48] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.318 second response time [04:56:50] RECOVERY - mw112 HTTPS on mw112 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.010 second response time [04:56:53] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [04:56:53] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [04:57:08] RECOVERY - prometheus101 APT on prometheus101 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [04:57:12] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.304 second response time [04:57:12] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.321 second response time [04:57:13] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.335 second response time [04:57:14] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [04:57:17] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [04:57:20] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.018 second response time [04:57:25] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.301 second response time [04:57:28] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [04:57:28] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.324 second response time [04:57:28] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.024 second response time [04:57:36] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.310 second response time [05:00:12] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 3% [05:00:15] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [05:00:33] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [05:00:46] RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 4% [05:00:46] RECOVERY - prometheus101 Current Load on prometheus101 is OK: OK - load average: 0.74, 1.06, 0.90 [05:00:53] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [05:00:53] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [05:01:01] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.98, 2.19, 1.93 [05:01:02] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 1% [05:18:05] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [05:18:15] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:18:28] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [05:18:33] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [05:18:34] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [05:18:35] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [05:18:36] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: HTTP CRITICAL - No data received from host [05:18:36] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:18:38] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:18:40] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:18:44] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [05:18:45] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [05:18:45] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:18:46] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:18:46] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [05:18:47] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:18:49] PROBLEM - mw101 HTTPS on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:18:50] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:18:52] PROBLEM - jobchron121 Current Load on jobchron121 is CRITICAL: CRITICAL - load average: 2.14, 1.20, 0.53 [05:18:53] PROBLEM - mw122 HTTPS on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:18:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 25.81, 13.26, 6.33 [05:19:02] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: HTTP CRITICAL - No data received from host [05:19:04] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:19:08] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:19:15] PROBLEM - mw102 HTTPS on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:19:32] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [05:19:40] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [05:19:54] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:20:15] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:20:20] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [05:20:20] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [05:20:24] PROBLEM - mw112 HTTPS on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:20:24] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:20:25] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [05:20:27] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 11.10, 4.49, 2.01 [05:20:27] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [05:20:27] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: HTTP CRITICAL - No data received from host [05:20:29] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:20:32] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:20:34] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:20:47] RECOVERY - mw101 HTTPS on mw101 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.008 second response time [05:21:01] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 55% [05:21:02] PROBLEM - gluster121 PowerDNS Recursor on gluster121 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:21:07] PROBLEM - mail121 Current Load on mail121 is CRITICAL: CRITICAL - load average: 6.07, 3.27, 1.42 [05:21:38] PROBLEM - cloud12 Current Load on cloud12 is CRITICAL: CRITICAL - load average: 51.77, 39.53, 24.40 [05:21:52] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [05:22:18] RECOVERY - mw112 HTTPS on mw112 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.011 second response time [05:22:26] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.424 second response time [05:22:26] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.719 second response time [05:22:30] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.715 second response time [05:22:30] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.062 second response time [05:22:31] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.313 second response time [05:22:32] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [05:22:33] PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 85% [05:22:33] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.020 second response time [05:22:34] PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 83% [05:22:38] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.321 second response time [05:22:39] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [05:22:39] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.018 second response time [05:22:43] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [05:22:43] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.323 second response time [05:22:44] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.019 second response time [05:22:48] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.017 second response time [05:22:48] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.320 second response time [05:22:48] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.317 second response time [05:22:50] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.240 second response time [05:22:51] PROBLEM - jobchron121 Current Load on jobchron121 is WARNING: WARNING - load average: 1.55, 1.74, 0.92 [05:22:52] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.310 second response time [05:22:54] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.272 second response time [05:22:55] RECOVERY - mw122 HTTPS on mw122 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.009 second response time [05:22:57] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [05:23:06] RECOVERY - gluster121 PowerDNS Recursor on gluster121 is OK: DNS OK: 4.679 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [05:23:10] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.266 second response time [05:23:11] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.025 second response time [05:23:15] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.250 second response time [05:23:21] RECOVERY - mw102 HTTPS on mw102 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.008 second response time [05:23:32] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.343 second response time [05:23:38] RECOVERY - cloud12 Current Load on cloud12 is OK: OK - load average: 18.90, 32.77, 23.88 [05:23:41] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.317 second response time [05:23:58] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.326 second response time [05:24:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 0.39, 0.79, 3.80 [05:24:20] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.313 second response time [05:24:21] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.019 second response time [05:24:24] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.316 second response time [05:24:28] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.281 second response time [05:24:30] RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 35% [05:24:50] RECOVERY - jobchron121 Current Load on jobchron121 is OK: OK - load average: 0.23, 1.18, 0.82 [05:25:01] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 64% [05:26:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 0.31, 0.62, 3.39 [05:26:20] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [05:26:20] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [05:26:27] RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 1% [05:27:00] PROBLEM - mail121 Current Load on mail121 is WARNING: WARNING - load average: 0.18, 1.95, 1.58 [05:27:02] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 2% [05:27:46] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [05:28:15] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [05:30:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/b06ea9233943 [05:30:54] [02miraheze/puppet] 07paladox 03b06ea92 - Install gluster102 and 122 [05:30:55] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [05:30:56] RECOVERY - mail121 Current Load on mail121 is OK: OK - load average: 0.19, 0.94, 1.24 [05:30:56] [url] Page not found · GitHub · GitHub | github.com [05:30:57] [02puppet] 07paladox opened pull request 03#2718: Install gluster102 and 122 - 13https://github.com/miraheze/puppet/pull/2718 [05:30:57] ... [05:31:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/b06ea9233943...91fcb38a2b75 [05:31:16] [url] Comparing b06ea9233943...91fcb38a2b75 · miraheze/puppet · GitHub | github.com [05:31:17] [02miraheze/puppet] 07paladox 0391fcb38 - Create gluster122.yaml [05:31:19] [02puppet] 07paladox synchronize pull request 03#2718: Install gluster102 and 122 - 13https://github.com/miraheze/puppet/pull/2718 [05:31:19] [url] Page not found · GitHub · GitHub | github.com [05:31:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/91fcb38a2b75...69090f7dad18 [05:31:40] [url] Comparing 91fcb38a2b75...69090f7dad18 · miraheze/puppet · GitHub | github.com [05:31:41] [02miraheze/puppet] 07paladox 0369090f7 - Update site.pp [05:31:43] [02puppet] 07paladox synchronize pull request 03#2718: Install gluster102 and 122 - 13https://github.com/miraheze/puppet/pull/2718 [05:31:43] [url] Page not found · GitHub · GitHub | github.com [05:34:49] [02puppet] 07paladox closed pull request 03#2718: Install gluster102 and 122 - 13https://github.com/miraheze/puppet/pull/2718 [05:34:49] [url] Page not found · GitHub · GitHub | github.com [05:34:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±1] 13https://github.com/miraheze/puppet/compare/96daf13c1c21...8697ab832923 [05:34:51] [url] Comparing 96daf13c1c21...8697ab832923 · miraheze/puppet · GitHub | github.com [05:34:52] [02miraheze/puppet] 07paladox 038697ab8 - Install gluster102 and 122 (#2718) [05:34:53] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [05:34:55] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [05:34:55] [url] Page not found · GitHub · GitHub | github.com [05:35:23] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 0.70, 1.14, 1.84 [05:37:21] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 0.58, 0.94, 1.68 [05:48:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.48, 1.97, 3.99 [05:51:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 2.76, 2.28 [05:52:30] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.16, 1.79, 1.27 [05:53:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.62, 3.14, 2.48 [05:53:50] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.39, 1.62, 3.20 [05:54:26] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.51, 1.37, 1.18 [05:55:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.79, 4.11, 2.91 [05:58:03] !log migrating gluster111 data [05:58:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:01:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.53, 3.48, 3.40 [06:11:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.26, 3.63, 2.66 [06:15:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.61, 3.65, 2.88 [06:17:14] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.71, 3.17, 2.79 [06:17:59] [02mw-config] 07Universal-Omega closed pull request 03#4832: Set actor table schema migration stage to WRITE_NEW in production - 13https://github.com/miraheze/mw-config/pull/4832 [06:18:00] [url] Page not found · GitHub · GitHub | github.com [06:18:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/480187ad3373...81bcb2c6dde8 [06:18:01] [url] Comparing 480187ad3373...81bcb2c6dde8 · miraheze/mw-config · GitHub | github.com [06:18:02] [02miraheze/mw-config] 07Universal-Omega 0381bcb2c - Set actor table schema migration stage to WRITE_NEW in production (#4832) [06:18:04] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [06:18:05] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [06:18:06] ... [06:18:19] [02MirahezeMagic] 07Universal-Omega closed pull request 03#347: Improve AssignImportedEdits - 13https://github.com/miraheze/MirahezeMagic/pull/347 [06:18:19] [url] Page not found · GitHub · GitHub | github.com [06:18:21] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/7678bd3a7fb0...8f6c75dfa871 [06:18:21] [url] Comparing 7678bd3a7fb0...8f6c75dfa871 · miraheze/MirahezeMagic · GitHub | github.com [06:18:22] [02miraheze/MirahezeMagic] 07Universal-Omega 038f6c75d - Improve AssignImportedEdits (#347) [06:18:23] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [06:18:26] [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/MirahezeMagic [06:18:26] [url] Page not found · GitHub · GitHub | github.com [06:19:06] miraheze/mw-config - Universal-Omega the build passed. [06:20:40] [02mediawiki] 07Universal-Omega commented on pull request 03#8334: Bump extensions/MirahezeMagic from `2a7be60` to `7678bd3` - 13https://github.com/miraheze/mediawiki/pull/8334#issuecomment-1200762402 [06:20:41] [url] Bump extensions/MirahezeMagic from `2a7be60` to `7678bd3` by dependabot[bot] · Pull Request #8334 · miraheze/mediawiki · GitHub | github.com [06:20:42] [02mediawiki] 07dependabot[bot] edited pull request 03#8334: Bump extensions/MirahezeMagic from `2a7be60` to `7678bd3` - 13https://github.com/miraheze/mediawiki/pull/8334 [06:20:42] [url] Page not found · GitHub · GitHub | github.com [06:21:35] [02mediawiki] 07dependabot[bot] edited pull request 03#8334: Bump extensions/MirahezeMagic from `2a7be60` to `7678bd3` - 13https://github.com/miraheze/mediawiki/pull/8334 [06:21:35] [url] Page not found · GitHub · GitHub | github.com [06:21:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/extensions/MirahezeMagic-8f6c75d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4ec4261d903b [06:21:42] [02miraheze/mediawiki] 07dependabot[bot] 034ec4261 - Bump extensions/MirahezeMagic from `2a7be60` to `8f6c75d` [06:21:43] [02mediawiki] 07dependabot[bot] opened pull request 03#8431: Bump extensions/MirahezeMagic from `2a7be60` to `8f6c75d` - 13https://github.com/miraheze/mediawiki/pull/8431 [06:21:44] [url] Page not found · GitHub · GitHub | github.com [06:21:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/extensions/MirahezeMagic-8f6c75d - 13https://github.com/miraheze/mediawiki [06:21:45] ... [06:21:46] [02mediawiki] 07dependabot[bot] commented on pull request 03#8334: Bump extensions/MirahezeMagic from `2a7be60` to `7678bd3` - 13https://github.com/miraheze/mediawiki/pull/8334#issuecomment-1200763281 [06:21:47] [url] Bump extensions/MirahezeMagic from `2a7be60` to `7678bd3` by dependabot[bot] · Pull Request #8334 · miraheze/mediawiki · GitHub | github.com [06:21:48] [02mediawiki] 07dependabot[bot] closed pull request 03#8334: Bump extensions/MirahezeMagic from `2a7be60` to `7678bd3` - 13https://github.com/miraheze/mediawiki/pull/8334 [06:21:48] [url] Page not found · GitHub · GitHub | github.com [06:21:49] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/MirahezeMagic-7678bd3 [06:21:51] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_38/extensions/MirahezeMagic-7678bd3 - 13https://github.com/miraheze/mediawiki [06:21:51] ... [06:22:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#8431: Bump extensions/MirahezeMagic from `2a7be60` to `8f6c75d` - 13https://github.com/miraheze/mediawiki/pull/8431 [06:22:07] [url] Page not found · GitHub · GitHub | github.com [06:22:08] [02mediawiki] 07github-actions[bot] labeled pull request 03#8431: Bump extensions/MirahezeMagic from `2a7be60` to `8f6c75d` - 13https://github.com/miraheze/mediawiki/pull/8431 [06:22:09] ... [06:23:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.66, 3.52, 3.07 [06:23:27] !log [@mwtask111] starting deploy of {'config': True} to all [06:23:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:23:48] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 23s [06:23:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:25:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.43, 3.90, 3.24 [06:26:52] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:26:52] miraheze/MirahezeMagic - Universal-Omega the build passed. [06:27:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.37, 3.46, 3.15 [06:27:14] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:27:30] PROBLEM - mail121 Current Load on mail121 is CRITICAL: CRITICAL - load average: 2.37, 2.31, 1.17 [06:29:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.72, 3.28, 3.13 [06:29:30] RECOVERY - mail121 Current Load on mail121 is OK: OK - load average: 0.40, 1.59, 1.05 [06:30:45] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:31:44] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:33:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.84, 3.57, 3.27 [06:34:08] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:34:17] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:39:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.12, 3.69, 3.40 [06:41:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.25, 3.65, 3.43 [06:41:20] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:46] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:42:19] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:42:20] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:42:21] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:42:37] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:42:44] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:43:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.46, 4.06, 3.61 [06:43:22] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 28480 bytes in 3.802 second response time [06:43:45] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.421 second response time [06:44:17] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.019 second response time [06:44:17] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.414 second response time [06:44:22] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.574 second response time [06:44:23] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.374 second response time [06:44:39] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 2.644 second response time [06:44:42] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [06:44:59] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.267 second response time [06:45:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.90, 3.66, 3.51 [06:45:37] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 9.661 second response time [06:46:42] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.534 second response time [06:47:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.02, 3.20, 3.37 [06:48:33] [02mediawiki] 07Reception123 closed pull request 03#8431: Bump extensions/MirahezeMagic from `2a7be60` to `8f6c75d` - 13https://github.com/miraheze/mediawiki/pull/8431 [06:48:34] [url] Page not found · GitHub · GitHub | github.com [06:48:35] [02miraheze/mediawiki] 07Reception123 deleted branch 03dependabot/submodules/REL1_38/extensions/MirahezeMagic-8f6c75d [06:48:36] [02mediawiki] 07Reception123 deleted branch 03dependabot/submodules/REL1_38/extensions/MirahezeMagic-8f6c75d - 13https://github.com/miraheze/mediawiki [06:48:37] [url] Page not found · GitHub · GitHub | github.com [06:48:38] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/dae8a8c5442a...c9097b8905f6 [06:48:38] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.197 second response time [06:48:39] [url] Comparing dae8a8c5442a...c9097b8905f6 · miraheze/mediawiki · GitHub | github.com [06:48:39] [02miraheze/mediawiki] 07dependabot[bot] 03c9097b8 - Bump extensions/MirahezeMagic from `2a7be60` to `8f6c75d` (#8431) [06:48:56] !log [reception@mwtask111] starting deploy of {'world': True, 'l10n': True} to all [06:49:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:51:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.06, 3.35, 3.41 [06:53:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.58, 3.01, 3.27 [06:54:38] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [06:55:38] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:55:44] !log [reception@mwtask111] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [06:55:52] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:56:41] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:56:51] !log [reception@mwtask111] starting deploy of {'world': True, 'l10n': True} to all [06:56:55] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:56:56] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:57:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:57:34] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.324 second response time [06:57:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:58:46] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.393 second response time [06:58:54] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [06:59:17] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:59:21] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK [06:59:44] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:01:07] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.041 second response time [07:01:16] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.968 second response time [07:01:49] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.332 second response time [07:02:01] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 4.130 second response time [07:03:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.47, 3.76, 3.53 [07:08:06] !log [reception@mwtask111] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 674s [07:08:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:08:28] Reception123: why aborted? [07:09:10] RhinosF1: bast connection issue [07:09:19] fatal: unable to access 'https://github.com/wikimedia/mediawiki-extensions-GlobalCssJs/': Failed to connect to bast.miraheze.org port 8080: Connection timed out [07:09:19] [url] GitHub - wikimedia/mediawiki-extensions-GlobalCssJs: Github mirror of MediaWiki extension GlobalCssJs - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_acce | github.com [07:09:24] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.71, 4.13, 3.71 [07:11:18] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.78, 4.00, 3.71 [07:13:27] Ok [07:17:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.46, 4.00, 3.72 [07:20:02] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:12] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:28] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:35] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:36] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:36] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:39] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:21:59] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.025 second response time [07:22:09] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.647 second response time [07:22:30] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.979 second response time [07:22:33] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.277 second response time [07:22:35] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.117 second response time [07:22:36] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.926 second response time [07:22:40] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.306 second response time [07:35:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.29, 3.90, 3.97 [07:37:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.34, 4.03, 4.00 [07:39:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.92, 3.93, 3.96 [07:39:38] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [07:39:43] RECOVERY - sbs.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'sbs.wiki' will expire on Thu 01 Sep 2022 00:42:07 GMT +0000. [07:41:35] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:50] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:42:03] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:42:17] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:43:31] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.858 second response time [07:43:45] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.313 second response time [07:44:01] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [07:44:17] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.572 second response time [07:45:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.08, 3.53, 3.76 [07:47:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.48, 3.71 [07:55:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.41, 3.53, 3.62 [08:01:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.78, 3.84, 3.84 [08:09:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.43, 2.81, 3.33 [08:11:20] PROBLEM - sbs.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address sbs.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:21:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.93, 3.74, 3.41 [08:23:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.31, 3.11, 3.21 [08:29:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.36, 3.80, 3.46 [08:31:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.75, 3.37, 3.34 [08:35:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.11, 3.43, 3.38 [08:37:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.43, 3.21, 3.31 [08:52:10] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.77, 1.60, 1.19 [08:53:54] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.52, 3.82, 3.28 [08:56:06] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.41, 1.64, 1.31 [09:03:24] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.52, 3.86, 3.67 [09:10:32] PROBLEM - wiki.leapit.no - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.leapit.no' expires in 15 day(s) (Wed 17 Aug 2022 08:40:55 GMT +0000). [09:11:51] PROBLEM - hoseki.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hoseki.wiki' expires in 15 day(s) (Wed 17 Aug 2022 08:43:28 GMT +0000). [09:12:11] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3d14c1e9f461...9fa820f4719f [09:12:11] [url] Comparing 3d14c1e9f461...9fa820f4719f · miraheze/ssl · GitHub | github.com [09:12:12] [02miraheze/ssl] 07MirahezeSSLBot 039fa820f - Bot: Update SSL cert for wiki.leapit.no [09:13:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.95, 4.08, 3.83 [09:17:37] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9fa820f4719f...842e9b9242ab [09:17:38] [url] Comparing 9fa820f4719f...842e9b9242ab · miraheze/ssl · GitHub | github.com [09:17:39] [02miraheze/ssl] 07MirahezeSSLBot 03842e9b9 - Bot: Update SSL cert for hoseki.wiki [09:26:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.21, 1.34, 1.91 [09:27:39] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [09:28:36] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.48, 1.73, 1.38 [09:30:34] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.81, 1.46, 1.32 [09:33:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.59, 3.80, 4.00 [09:35:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.33, 3.95, 4.03 [09:36:03] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.66, 1.39, 1.67 [09:37:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.79, 3.96 [09:40:16] RECOVERY - wiki.leapit.no - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.leapit.no' will expire on Sun 30 Oct 2022 08:12:03 GMT +0000. [09:41:40] RECOVERY - hoseki.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hoseki.wiki' will expire on Sun 30 Oct 2022 08:17:32 GMT +0000. [09:43:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.02, 3.73, 3.84 [09:45:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.14, 3.16, 3.62 [09:47:39] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [09:49:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.89, 2.52, 3.25 [09:52:57] PROBLEM - roblox-wiki.tk - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query roblox-wiki.tk. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:54:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.26, 1.79, 1.61 [09:59:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.20, 3.40, 3.22 [10:01:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.60, 3.05, 3.11 [10:07:06] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.07, 1.72, 1.36 [10:07:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.46, 3.26, 3.15 [10:09:04] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.38, 1.65, 1.38 [10:09:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.29, 3.29, 3.18 [10:17:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.52, 3.52, 3.33 [10:20:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.30, 10.03, 9.23 [10:21:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.67, 3.83, 3.48 [10:21:33] RECOVERY - roblox-wiki.tk - reverse DNS on sslhost is OK: SSL OK - roblox-wiki.tk reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK [10:22:07] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.42, 10.86, 9.64 [10:24:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.96, 10.45, 9.61 [10:25:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.36, 3.43, 3.39 [10:29:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.86, 3.96, 3.60 [10:30:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.78, 1.89, 1.97 [10:31:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.28, 3.70, 3.55 [10:31:58] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.28, 10.04, 9.87 [10:33:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.47, 4.19, 3.75 [10:34:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.08, 1.90, 1.95 [10:35:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.21, 3.89, 3.69 [10:36:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.75, 1.81, 1.91 [10:37:42] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.60, 1.71, 1.43 [10:39:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.08, 2.99, 3.37 [10:39:41] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.95, 1.70, 1.46 [10:41:39] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.14, 1.50, 1.42 [10:45:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.78, 3.61, 3.53 [10:47:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.57, 4.00, 3.68 [10:49:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.85, 3.77, 3.66 [10:50:03] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.55, 1.56, 1.70 [10:51:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.40, 9.34, 9.15 [10:53:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.25, 9.00, 9.06 [10:55:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.01, 2.96, 3.35 [11:05:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.41, 3.34, 3.33 [11:07:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.57, 3.07, 3.23 [11:12:25] [02miraheze/ErrorPages] 07translatewiki pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/c5ee580377f0...174bab9a9716 [11:12:26] [url] Comparing c5ee580377f0...174bab9a9716 · miraheze/ErrorPages · GitHub | github.com [11:12:27] [02miraheze/ErrorPages] 07translatewiki 03174bab9 - Localisation updates from https://translatewiki.net. [11:12:27] [url] Main page - translatewiki.net | translatewiki.net [11:12:28] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/CreateWiki/compare/0a62452a6a5f...3d7f9ec546cf [11:12:29] [url] Comparing 0a62452a6a5f...3d7f9ec546cf · miraheze/CreateWiki · GitHub | github.com [11:12:30] [02miraheze/CreateWiki] 07translatewiki 033d7f9ec - Localisation updates from https://translatewiki.net. [11:12:30] [url] Main page - translatewiki.net | translatewiki.net [11:12:31] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/aad10605e817...eb7cb95641f6 [11:12:32] [url] Comparing aad10605e817...eb7cb95641f6 · miraheze/ManageWiki · GitHub | github.com [11:12:33] [02miraheze/ManageWiki] 07translatewiki 03eb7cb95 - Localisation updates from https://translatewiki.net. [11:12:33] [url] Main page - translatewiki.net | translatewiki.net [11:12:34] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/8f6c75dfa871...8fc0344525cd [11:12:35] [url] Comparing 8f6c75dfa871...8fc0344525cd · miraheze/MirahezeMagic · GitHub | github.com [11:12:36] [02miraheze/MirahezeMagic] 07translatewiki 038fc0344 - Localisation updates from https://translatewiki.net. [11:12:36] [url] Main page - translatewiki.net | translatewiki.net [11:15:17] miraheze/ErrorPages - translatewiki the build passed. [11:18:08] miraheze/ManageWiki - translatewiki the build passed. [11:22:05] miraheze/CreateWiki - translatewiki the build passed. [11:27:34] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [11:27:36] miraheze/MirahezeMagic - translatewiki the build passed. [11:27:59] PROBLEM - history.estill.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:59] PROBLEM - stablestate.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:03] PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:19] PROBLEM - test3.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:27] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 3 backends are down. mw101 mw111 mw122 [11:29:34] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:30:00] RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3407 bytes in 0.017 second response time [11:30:12] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:39] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 5 backends are down. mw102 mw111 mw112 mw121 mw122 [11:31:16] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:31:46] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:32:09] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [11:32:39] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [11:33:15] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.505 second response time [11:33:41] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 1.121 second response time [11:34:46] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:11] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [11:35:38] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:48] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:36:43] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:37:57] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.014 second response time [11:38:40] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.061 second response time [11:38:50] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.303 second response time [11:39:23] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.19, 1.78, 1.52 [11:39:24] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.55, 3.50, 3.19 [11:41:18] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.37, 3.12, 3.09 [11:41:55] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.223 second response time [11:43:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.65, 8.82, 7.70 [11:45:00] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.56, 8.21, 7.61 [11:45:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.12, 3.45, 3.26 [11:47:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.40, 3.05, 3.13 [11:53:47] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.80, 1.58, 1.37 [11:55:05] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:46] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.48, 1.93, 1.52 [11:56:33] RECOVERY - history.estill.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.estill.org' will expire on Wed 24 Aug 2022 03:29:39 GMT +0000. [11:56:33] RECOVERY - stablestate.org - LetsEncrypt on sslhost is OK: OK - Certificate 'stablestate.org' will expire on Tue 23 Aug 2022 08:41:48 GMT +0000. [11:57:01] RECOVERY - test3.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [11:57:03] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [11:57:44] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.92, 1.57, 1.44 [11:57:54] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.35, 3.45, 3.33 [11:59:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.32, 3.18, 3.23 [12:01:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.02, 9.79, 9.00 [12:01:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.98, 10.21, 9.34 [12:03:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.57, 9.64, 9.05 [12:05:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.29, 11.50, 10.03 [12:07:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.07, 10.51, 9.85 [12:08:13] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:14] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:34] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:43] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:44] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:09:22] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.79, 9.67, 9.62 [12:09:26] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 8.311 second response time [12:10:06] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:10] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:13] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:18] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:50] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.981 second response time [12:11:23] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.406 second response time [12:12:08] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.687 second response time [12:12:09] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.365 second response time [12:12:14] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 4.510 second response time [12:12:17] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.794 second response time [12:12:17] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 2.615 second response time [12:12:20] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.979 second response time [12:14:59] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.021 second response time [12:15:16] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:15:45] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:06] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.024 second response time [12:17:21] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.436 second response time [12:17:49] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.255 second response time [12:20:13] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.51, 10.96, 10.18 [12:20:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.67, 1.91, 1.99 [12:22:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.54, 11.13, 10.34 [12:28:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.01, 11.97, 10.93 [12:30:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.06, 11.38, 10.85 [12:33:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.71, 2.37, 1.71 [12:33:13] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:33:29] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.36, 4.45, 3.74 [12:35:10] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.98, 1.80, 1.57 [12:35:32] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:39] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:08] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.87, 1.56, 1.51 [12:37:45] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.159 second response time [12:38:03] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.27, 1.42, 1.66 [12:41:00] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.44, 9.22, 10.04 [12:44:20] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:28] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.521 second response time [12:47:18] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.618 second response time [12:47:54] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.116 second response time [12:47:56] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.939 second response time [12:48:15] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.443 second response time [12:49:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.26, 11.18, 10.48 [12:51:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.47, 3.85, 3.99 [12:51:41] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.66, 10.80, 10.44 [12:53:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.69, 3.97, 4.01 [12:53:10] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [12:53:39] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:39] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.10, 11.27, 10.66 [12:54:48] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.67, 3.88 [12:55:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.18, 11.38, 10.78 [12:56:58] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.54, 4.03, 3.99 [12:57:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.43, 11.79, 10.98 [12:57:52] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.562 second response time [12:58:38] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:59:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.11, 11.59, 11.02 [13:01:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.20, 3.76, 3.91 [13:02:01] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.73, 1.78, 1.25 [13:02:13] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:02:56] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:58] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.70, 1.36, 1.16 [13:04:46] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 3.390 second response time [13:04:54] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.263 second response time [13:05:25] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.888 second response time [13:05:32] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.877 second response time [13:06:17] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.021 second response time [13:06:38] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.55, 1.77, 1.61 [13:08:36] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.05, 1.63, 1.59 [13:11:22] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.43, 8.83, 10.16 [13:15:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 3.38, 3.49 [13:17:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.06, 3.29, 3.45 [13:19:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.33, 3.05, 3.34 [13:22:26] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.07, 1.77, 1.51 [13:28:10] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [13:30:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.86, 1.99, 1.75 [13:30:30] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:56] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:02] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:03] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.011 second response time [13:31:13] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:47] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:32:32] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.594 second response time [13:32:56] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.135 second response time [13:32:59] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.336 second response time [13:32:59] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.024 second response time [13:33:12] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.414 second response time [13:33:55] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.250 second response time [13:34:02] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.03, 10.45, 9.98 [13:34:09] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.21, 2.05, 1.83 [13:36:00] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.40, 10.13, 9.93 [13:41:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.18, 3.59, 3.35 [13:43:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.87, 3.37, 3.29 [13:47:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.92, 3.64, 3.41 [13:49:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.28, 3.99, 3.57 [13:50:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.73, 10.62, 10.03 [13:51:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.01, 3.77, 3.55 [13:53:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.77, 4.20, 3.74 [13:54:39] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.02, 10.84, 10.22 [13:56:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.43, 10.61, 10.21 [13:57:54] [02miraheze/ImportDump] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.1 [+0/-0/±2] 13https://github.com/miraheze/ImportDump/commit/4e983bfb99bd [13:57:55] [02miraheze/ImportDump] 07dependabot[bot] 034e983bf - Bump stylelint-config-wikimedia from 0.13.0 to 0.13.1 [13:57:57] [02ImportDump] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.1 - 13https://github.com/miraheze/ImportDump [13:57:57] [url] Page not found · GitHub · GitHub | github.com [13:57:58] [02ImportDump] 07dependabot[bot] opened pull request 03#22: Bump stylelint-config-wikimedia from 0.13.0 to 0.13.1 - 13https://github.com/miraheze/ImportDump/pull/22 [13:57:59] [url] Page not found · GitHub · GitHub | github.com [13:58:00] [02ImportDump] 07dependabot[bot] labeled pull request 03#22: Bump stylelint-config-wikimedia from 0.13.0 to 0.13.1 - 13https://github.com/miraheze/ImportDump/pull/22 [13:58:00] [url] Page not found · GitHub · GitHub | github.com [13:58:01] [02ImportDump] 07dependabot[bot] labeled pull request 03#22: Bump stylelint-config-wikimedia from 0.13.0 to 0.13.1 - 13https://github.com/miraheze/ImportDump/pull/22 [13:58:02] [url] Page not found · GitHub · GitHub | github.com [13:58:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.84, 10.16, 10.10 [14:02:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.86, 1.89, 1.98 [14:06:13] miraheze/ImportDump - dependabot[bot] the build passed. [14:06:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.21, 10.42, 10.27 [14:08:08] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [14:10:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.58, 9.77, 10.04 [14:12:17] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:22] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:05] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:06] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:16:24] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.947 second response time [14:16:35] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.135 second response time [14:17:12] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 6.775 second response time [14:17:19] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.744 second response time [14:22:03] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.27, 1.43, 1.65 [14:28:10] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:28:20] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:28:31] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:28:35] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:29:10] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:19] [02miraheze/mediawiki] 07MacFan4000 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7ed52f7fdf2b [14:31:20] [02miraheze/mediawiki] 07dependabot[bot] 037ed52f7 - Bump skins/chameleon from `a923c46` to `e08fffc` [14:31:22] [02mediawiki] 07MacFan4000 created branch 03REL1_37 - 13https://github.com/miraheze/mediawiki [14:31:22] [url] Page not found · GitHub · GitHub | github.com [14:31:45] [02miraheze/mediawiki] 07MacFan4000 created branch 03dependabot/submodules/REL1_37/skins/chameleon-e08fffc 13https://github.com/miraheze/mediawiki/commit/7ed52f7fdf2b633b696386160a644f1811bc9d51 [14:31:47] [02mediawiki] 07MacFan4000 created branch 03dependabot/submodules/REL1_37/skins/chameleon-e08fffc - 13https://github.com/miraheze/mediawiki [14:31:48] ... [14:34:38] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.795 second response time [14:34:38] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.366 second response time [14:35:31] [02miraheze/mediawiki] 07MacFan4000 deleted branch 03REL1_37 [14:35:32] [02mediawiki] 07MacFan4000 deleted branch 03REL1_37 - 13https://github.com/miraheze/mediawiki [14:35:33] [url] Page not found · GitHub · GitHub | github.com [14:35:35] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.092 second response time [14:36:31] [02miraheze/mediawiki] 07MacFan4000 created branch 03REL1_37 13https://github.com/miraheze/mediawiki/commit/7ed52f7fdf2b633b696386160a644f1811bc9d51 [14:36:33] [02mediawiki] 07MacFan4000 created branch 03REL1_37 - 13https://github.com/miraheze/mediawiki [14:36:33] ... [14:36:46] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 3.786 second response time [14:36:50] [02miraheze/mediawiki] 07MacFan4000 deleted branch 03REL1_37 [14:36:52] [02mediawiki] 07MacFan4000 deleted branch 03REL1_37 - 13https://github.com/miraheze/mediawiki [14:36:52] ... [14:36:52] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.916 second response time [14:37:09] [02miraheze/mediawiki] 07MacFan4000 created branch 03REL1_37 13https://github.com/miraheze/mediawiki/commit/2035d98f38d59f0b6650a09345d8578c6f941415 [14:37:10] [02mediawiki] 07MacFan4000 created branch 03REL1_37 - 13https://github.com/miraheze/mediawiki [14:37:31] [02mediawiki] 07MacFan4000 reopened pull request 03#5522: Bump skins/chameleon from `a923c46` to `e08fffc` - 13https://github.com/miraheze/mediawiki/pull/5522 [14:37:49] [02mediawiki] 07MacFan4000 commented on pull request 03#5522: Bump skins/chameleon from `a923c46` to `e08fffc` - 13https://github.com/miraheze/mediawiki/pull/5522#issuecomment-1201292317 [14:37:50] [url] Bump skins/chameleon from `a923c46` to `e08fffc` by dependabot[bot] · Pull Request #5522 · miraheze/mediawiki · GitHub | github.com [14:37:52] [02mediawiki] 07dependabot[bot] edited pull request 03#5522: Bump skins/chameleon from `a923c46` to `e08fffc` - 13https://github.com/miraheze/mediawiki/pull/5522 [14:37:52] [url] Page not found · GitHub · GitHub | github.com [14:38:43] [02mediawiki] 07dependabot[bot] edited pull request 03#5522: Bump skins/chameleon from `a923c46` to `e08fffc` - 13https://github.com/miraheze/mediawiki/pull/5522 [14:38:43] [url] Page not found · GitHub · GitHub | github.com [14:38:47] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_38/skins/chameleon-783da49 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/47cfd9e276ea [14:38:49] [02miraheze/mediawiki] 07dependabot[bot] 0347cfd9e - Bump skins/chameleon from `f78f24b` to `783da49` [14:38:50] [02mediawiki] 07dependabot[bot] opened pull request 03#8432: Bump skins/chameleon from `f78f24b` to `783da49` - 13https://github.com/miraheze/mediawiki/pull/8432 [14:38:51] [url] Page not found · GitHub · GitHub | github.com [14:38:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_38/skins/chameleon-783da49 - 13https://github.com/miraheze/mediawiki [14:38:53] [url] Page not found · GitHub · GitHub | github.com [14:39:17] [02mediawiki] 07github-actions[bot] labeled pull request 03#8432: Bump skins/chameleon from `f78f24b` to `783da49` - 13https://github.com/miraheze/mediawiki/pull/8432 [14:39:18] ... [14:39:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#8432: Bump skins/chameleon from `f78f24b` to `783da49` - 13https://github.com/miraheze/mediawiki/pull/8432 [14:39:19] [url] Page not found · GitHub · GitHub | github.com [14:39:57] CosmicAlpha: fixed ^^ [14:40:20] [02mediawiki] 07MacFan4000 closed pull request 03#5522: Bump skins/chameleon from `a923c46` to `e08fffc` - 13https://github.com/miraheze/mediawiki/pull/5522 [14:40:21] ... [14:40:23] [02mediawiki] 07dependabot[bot] commented on pull request 03#5522: Bump skins/chameleon from `a923c46` to `e08fffc` - 13https://github.com/miraheze/mediawiki/pull/5522#issuecomment-1201295091 [14:40:23] [url] Bump skins/chameleon from `a923c46` to `e08fffc` by dependabot[bot] · Pull Request #5522 · miraheze/mediawiki · GitHub | github.com [14:40:32] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/skins/chameleon-e08fffc - 13https://github.com/miraheze/mediawiki [14:40:33] ... [14:40:34] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/skins/chameleon-e08fffc [14:41:16] [02miraheze/mediawiki] 07MacFan4000 deleted branch 03REL1_37 [14:41:17] [02mediawiki] 07MacFan4000 deleted branch 03REL1_37 - 13https://github.com/miraheze/mediawiki [14:41:17] [url] Page not found · GitHub · GitHub | github.com [14:41:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.81, 9.61, 8.82 [14:42:04] I temp recreated REL1_37 [14:43:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.90, 9.84, 9.02 [14:49:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.81, 1.74, 1.23 [14:53:15] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.27, 1.67, 1.33 [14:57:30] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:57:31] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.83, 10.38, 9.48 [14:59:30] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 2.367 second response time [14:59:34] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.129 second response time [15:01:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.95, 10.05, 9.47 [15:03:06] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:47] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:12] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:16] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.015 second response time [15:06:18] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.26, 1.78, 1.53 [15:06:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.11, 10.56, 9.69 [15:06:29] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:47] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.186 second response time [15:12:17] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.60, 10.12, 9.81 [15:13:15] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.824 second response time [15:13:25] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28480 bytes in 3.146 second response time [15:14:30] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.359 second response time [15:14:37] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.766 second response time [15:22:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.33, 10.23, 9.88 [15:22:06] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.48, 10.03, 9.66 [15:24:00] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.13, 9.98, 9.70 [15:24:04] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.84, 9.55, 9.66 [15:26:20] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.94, 1.83, 1.60 [15:27:57] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.94, 3.20, 2.90 [15:27:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.51, 10.21, 9.91 [15:28:08] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [15:28:19] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.11, 1.55, 1.52 [15:31:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.90, 3.58, 3.99 [15:31:57] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.57, 3.01, 2.90 [15:33:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.80, 9.79, 9.80 [15:34:26] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.05, 10.37, 9.94 [15:35:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.86, 4.41, 4.20 [15:36:20] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.30, 8.97, 9.50 [15:40:10] PROBLEM - largedu.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'largedu.eu.org' expires in 15 day(s) (Wed 17 Aug 2022 15:37:36 GMT +0000). [15:40:41] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:41:37] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:42:43] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.459 second response time [15:43:38] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 4.616 second response time [15:46:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/842e9b9242ab...5a5d651d6137 [15:46:47] [url] Comparing 842e9b9242ab...5a5d651d6137 · miraheze/ssl · GitHub | github.com [15:46:48] [02miraheze/ssl] 07MirahezeSSLBot 035a5d651 - Bot: Update SSL cert for largedu.eu.org [15:47:38] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:39] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.720 second response time [15:49:43] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:44] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:55] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:02] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:03] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:50:16] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:51:17] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:51:21] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.68, 1.94, 2.00 [15:55:49] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:01] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:58:11] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:58:21] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:08] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [15:59:54] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.027 second response time [16:00:09] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.543 second response time [16:00:19] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.335 second response time [16:00:19] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 8.524 second response time [16:00:22] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.074 second response time [16:00:27] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.997 second response time [16:00:34] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.777 second response time [16:00:36] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 8.969 second response time [16:02:14] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [16:02:29] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.095 second response time [16:04:03] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.09, 1.34, 1.66 [16:04:08] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [16:06:13] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.890 second response time [16:06:15] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.047 second response time [16:09:49] RECOVERY - largedu.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'largedu.eu.org' will expire on Sun 30 Oct 2022 14:46:40 GMT +0000. [16:20:10] PROBLEM - mw112 APT on mw112 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:20:11] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:20:53] PROBLEM - db121 APT on db121 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (3 critical updates). [16:21:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.64, 3.66, 3.94 [16:21:09] PROBLEM - cp31 APT on cp31 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:21:36] PROBLEM - mem101 APT on mem101 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (3 critical updates). [16:22:07] PROBLEM - cp30 APT on cp30 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:22:21] PROBLEM - cloud11 APT on cloud11 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (3 critical updates). [16:22:22] PROBLEM - cloud13 APT on cloud13 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:22:29] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:22:32] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.76, 1.62, 1.33 [16:23:14] PROBLEM - cloud14 APT on cloud14 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:23:56] PROBLEM - prometheus101 APT on prometheus101 is CRITICAL: APT CRITICAL: 25 packages available for upgrade (3 critical updates). [16:24:07] PROBLEM - db101 APT on db101 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (3 critical updates). [16:24:31] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.06, 1.50, 1.33 [16:25:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.29, 3.91, 3.95 [16:25:12] PROBLEM - puppet111 APT on puppet111 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:25:41] PROBLEM - cp20 APT on cp20 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:27:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.06, 3.59, 3.83 [16:27:09] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.58, 1.61, 1.13 [16:27:23] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:28:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.98, 1.72, 1.58 [16:28:13] [02miraheze/MatomoAnalytics] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.1 [+0/-0/±2] 13https://github.com/miraheze/MatomoAnalytics/commit/2f65c81f79a0 [16:28:15] [02miraheze/MatomoAnalytics] 07dependabot[bot] 032f65c81 - Bump stylelint-config-wikimedia from 0.13.0 to 0.13.1 [16:28:16] [02MatomoAnalytics] 07dependabot[bot] opened pull request 03#82: Bump stylelint-config-wikimedia from 0.13.0 to 0.13.1 - 13https://github.com/miraheze/MatomoAnalytics/pull/82 [16:28:17] [url] Page not found · GitHub · GitHub | github.com [16:28:18] [02MatomoAnalytics] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.1 - 13https://github.com/miraheze/MatomoAnalytics [16:28:18] [url] Page not found · GitHub · GitHub | github.com [16:28:19] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#82: Bump stylelint-config-wikimedia from 0.13.0 to 0.13.1 - 13https://github.com/miraheze/MatomoAnalytics/pull/82 [16:28:20] [url] Page not found · GitHub · GitHub | github.com [16:28:21] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#82: Bump stylelint-config-wikimedia from 0.13.0 to 0.13.1 - 13https://github.com/miraheze/MatomoAnalytics/pull/82 [16:28:21] [url] Page not found · GitHub · GitHub | github.com [16:28:32] [02miraheze/MatomoAnalytics] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 [+0/-0/±2] 13https://github.com/miraheze/MatomoAnalytics/commit/4d99dd9d45f5 [16:28:33] [02miraheze/MatomoAnalytics] 07dependabot[bot] 034d99dd9 - Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 [16:28:35] [02MatomoAnalytics] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 - 13https://github.com/miraheze/MatomoAnalytics [16:28:35] ... [16:28:36] [02MatomoAnalytics] 07dependabot[bot] opened pull request 03#83: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/83 [16:28:37] ... [16:28:38] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#83: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/83 [16:28:38] ... [16:28:39] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#83: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/83 [16:28:40] [url] Page not found · GitHub · GitHub | github.com [16:28:46] PROBLEM - es131 APT on es131 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [16:28:59] PROBLEM - cp21 APT on cp21 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:29:05] PROBLEM - knifepointhorror.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'knifepointhorror.wiki' expires in 15 day(s) (Wed 17 Aug 2022 16:20:53 GMT +0000). [16:29:09] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.43, 1.57, 1.18 [16:30:03] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.24, 1.54, 1.53 [16:30:47] miraheze/MatomoAnalytics - dependabot[bot] the build has errored. [16:31:10] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/4d99dd9d45f5...690a97b1a77e [16:31:10] [url] Comparing 4d99dd9d45f5...690a97b1a77e · miraheze/MatomoAnalytics · GitHub | github.com [16:31:11] [02miraheze/MatomoAnalytics] 07Universal-Omega 03690a97b - Update .stylelintrc.json [16:31:13] [02MatomoAnalytics] 07Universal-Omega synchronize pull request 03#83: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/83 [16:31:13] [url] Page not found · GitHub · GitHub | github.com [16:31:50] miraheze/MatomoAnalytics - dependabot[bot] the build passed. [16:31:56] PROBLEM - gluster102 APT on gluster102 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:32:04] [02MatomoAnalytics] 07Universal-Omega closed pull request 03#82: Bump stylelint-config-wikimedia from 0.13.0 to 0.13.1 - 13https://github.com/miraheze/MatomoAnalytics/pull/82 [16:32:05] [url] Page not found · GitHub · GitHub | github.com [16:32:06] PROBLEM - mw101 APT on mw101 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:32:06] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MatomoAnalytics/compare/1260bb116711...452c45c16e30 [16:32:06] [url] Comparing 1260bb116711...452c45c16e30 · miraheze/MatomoAnalytics · GitHub | github.com [16:32:07] [02miraheze/MatomoAnalytics] 07dependabot[bot] 03452c45c - Bump stylelint-config-wikimedia from 0.13.0 to 0.13.1 (#82) [16:32:09] [02miraheze/MatomoAnalytics] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.1 [16:32:10] [02MatomoAnalytics] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.1 - 13https://github.com/miraheze/MatomoAnalytics [16:32:11] [url] Page not found · GitHub · GitHub | github.com [16:33:54] PROBLEM - es141 APT on es141 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [16:34:07] PROBLEM - db112 APT on db112 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (3 critical updates). [16:34:12] PROBLEM - mem121 APT on mem121 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (3 critical updates). [16:34:23] [02ImportDump] 07Universal-Omega closed pull request 03#22: Bump stylelint-config-wikimedia from 0.13.0 to 0.13.1 - 13https://github.com/miraheze/ImportDump/pull/22 [16:34:23] [url] Page not found · GitHub · GitHub | github.com [16:34:24] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ImportDump/compare/ab3032b43b45...085729daab56 [16:34:25] [url] Comparing ab3032b43b45...085729daab56 · miraheze/ImportDump · GitHub | github.com [16:34:26] [02miraheze/ImportDump] 07dependabot[bot] 03085729d - Bump stylelint-config-wikimedia from 0.13.0 to 0.13.1 (#22) [16:34:27] [02miraheze/ImportDump] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.1 [16:34:28] miraheze/MatomoAnalytics - Universal-Omega the build passed. [16:34:29] [02ImportDump] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.1 - 13https://github.com/miraheze/ImportDump [16:34:29] ... [16:35:17] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5a5d651d6137...7955c7f70c75 [16:35:17] [url] Comparing 5a5d651d6137...7955c7f70c75 · miraheze/ssl · GitHub | github.com [16:35:18] [02miraheze/ssl] 07MirahezeSSLBot 037955c7f - Bot: Update SSL cert for knifepointhorror.wiki [16:35:20] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/MatomoAnalytics/compare/452c45c16e30...57913819eb04 [16:35:21] [url] Comparing 452c45c16e30...57913819eb04 · miraheze/MatomoAnalytics · GitHub | github.com [16:35:22] [02miraheze/MatomoAnalytics] 07dependabot[bot] 035791381 - Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 (#83) [16:35:23] [02MatomoAnalytics] 07Universal-Omega closed pull request 03#83: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/83 [16:35:24] [url] Page not found · GitHub · GitHub | github.com [16:35:25] [02miraheze/MatomoAnalytics] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 [16:35:26] [02MatomoAnalytics] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 - 13https://github.com/miraheze/MatomoAnalytics [16:35:27] [url] Page not found · GitHub · GitHub | github.com [16:36:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.71, 10.23, 9.44 [16:37:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.89, 3.56, 3.57 [16:37:08] PROBLEM - matomo101 Current Load on matomo101 is CRITICAL: CRITICAL - load average: 4.04, 3.90, 3.31 [16:37:42] miraheze/MatomoAnalytics - Universal-Omega the build passed. [16:38:53] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.11, 9.91, 9.40 [16:39:05] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.21, 3.60, 3.27 [16:39:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.64, 3.42, 3.51 [16:39:50] miraheze/MatomoAnalytics - Universal-Omega the build passed. [16:40:04] PROBLEM - db111 APT on db111 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (3 critical updates). [16:41:03] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.49, 3.19, 3.16 [16:41:38] PROBLEM - mw102 APT on mw102 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:42:08] PROBLEM - cloud12 APT on cloud12 is CRITICAL: APT CRITICAL: 61 packages available for upgrade (3 critical updates). [16:42:16] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:35] miraheze/ImportDump - Universal-Omega the build passed. [16:42:49] PROBLEM - cloud10 APT on cloud10 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (3 critical updates). [16:43:30] PROBLEM - gluster122 APT on gluster122 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:44:09] PROBLEM - ldap111 APT on ldap111 is CRITICAL: APT CRITICAL: 24 packages available for upgrade (3 critical updates). [16:46:24] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.244 second response time [16:47:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.01, 3.17, 3.38 [16:50:41] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.02, 1.71, 1.55 [16:51:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.85, 3.99, 3.65 [16:53:49] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:30] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:38] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:53] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:53] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.89, 3.55, 3.56 [16:55:50] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.866 second response time [16:56:32] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.87, 1.98, 1.74 [16:56:38] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.716 second response time [16:56:47] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.292 second response time [16:56:57] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 9.054 second response time [16:56:57] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.774 second response time [16:57:48] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=eskomisanwiki --no-updates --username-prefix=wikia:komisan /mnt/mediawiki-static/metawiki/ImportDump/eskomisanwiki-20220801142035.xml (START) [16:58:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:58:54] RECOVERY - knifepointhorror.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'knifepointhorror.wiki' will expire on Sun 30 Oct 2022 15:35:11 GMT +0000. [17:00:48] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=eskomisanwiki --no-updates --username-prefix=wikia:komisan /mnt/mediawiki-static/metawiki/ImportDump/eskomisanwiki-20220801142035.xml (END - exit=0) [17:00:49] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=eskomisanwiki (START) [17:02:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:02:14] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=eskomisanwiki (END - exit=0) [17:02:15] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=eskomisanwiki --active --update (END - exit=0) [17:03:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:03:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.14, 2.87, 3.28 [17:03:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:04:10] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:20] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:32] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:04:43] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:05:00] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:18] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.14, 1.54, 1.68 [17:06:20] PROBLEM - prometheus101 Current Load on prometheus101 is CRITICAL: CRITICAL - load average: 6.65, 4.68, 2.17 [17:06:40] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:07:27] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:08:35] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.334 second response time [17:08:39] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:08:45] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:08:45] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:09:14] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [17:09:31] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.111 second response time [17:10:09] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.37, 1.84, 1.76 [17:10:53] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:10] PROBLEM - prometheus101 Current Load on prometheus101 is WARNING: WARNING - load average: 0.54, 1.75, 1.62 [17:12:43] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.022 second response time [17:12:46] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 1.648 second response time [17:12:47] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.788 second response time [17:12:52] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [17:13:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.87, 3.59, 3.49 [17:14:07] RECOVERY - prometheus101 Current Load on prometheus101 is OK: OK - load average: 0.16, 1.20, 1.43 [17:15:05] !log change retention period on graylog121 to 5 days [17:15:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:16:08] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:16:11] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:16:16] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:16:18] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:16:28] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:16:46] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:04] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.46, 4.16, 3.75 [17:17:08] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:09] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:19] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:21] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.744 second response time [17:17:42] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:18] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.252 second response time [17:18:22] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.177 second response time [17:18:25] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.830 second response time [17:18:56] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:57] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:06] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.17, 3.77, 3.66 [17:20:10] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:20:20] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.915 second response time [17:20:45] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 9.153 second response time [17:20:53] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.788 second response time [17:20:58] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 6.867 second response time [17:20:59] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.983 second response time [17:21:04] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.710 second response time [17:21:42] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.69, 3.29, 1.99 [17:21:47] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:07] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.332 second response time [17:22:13] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.337 second response time [17:24:23] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/3214e40ecbcb [17:24:25] [02miraheze/dns] 07paladox 033214e40 - Add prometheus131 to dns [17:24:26] [02dns] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/dns [17:24:27] [url] Page not found · GitHub · GitHub | github.com [17:24:28] [02dns] 07paladox opened pull request 03#311: Add prometheus131 to dns - 13https://github.com/miraheze/dns/pull/311 [17:24:28] [url] Page not found · GitHub · GitHub | github.com [17:24:56] [02dns] 07paladox closed pull request 03#311: Add prometheus131 to dns - 13https://github.com/miraheze/dns/pull/311 [17:24:56] [url] Page not found · GitHub · GitHub | github.com [17:24:57] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/4873046a7804...e4f0ffa6d178 [17:24:58] [url] Comparing 4873046a7804...e4f0ffa6d178 · miraheze/dns · GitHub | github.com [17:24:59] [02miraheze/dns] 07paladox 03e4f0ffa - Add prometheus131 to dns (#311) [17:25:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.57, 4.15, 3.81 [17:25:21] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:22] [02mw-config] 07Universal-Omega closed pull request 03#4816: Rename `$wgCosmosRailBlacklistedNamespaces` - 13https://github.com/miraheze/mw-config/pull/4816 [17:25:23] ... [17:25:24] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [17:25:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/81bcb2c6dde8...139e8c207b47 [17:25:26] [url] Comparing 81bcb2c6dde8...139e8c207b47 · miraheze/mw-config · GitHub | github.com [17:25:27] [02miraheze/mw-config] 07Universal-Omega 03139e8c2 - Rename `$wgCosmosRailBlacklistedNamespaces` (#4816) [17:25:28] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [17:25:29] ... [17:25:42] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.78, 1.90, 1.71 [17:25:46] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [17:25:47] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-3 [17:25:49] [02dns] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/dns [17:25:49] [url] Page not found · GitHub · GitHub | github.com [17:26:26] miraheze/mw-config - Universal-Omega the build passed. [17:26:30] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:26:31] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:26:39] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.74, 3.50, 3.61 [17:27:22] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 95s [17:27:23] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.660 second response time [17:27:42] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.14, 1.65, 1.64 [17:28:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.95, 2.00, 1.98 [17:28:14] !log UPDATE mw_namespaces SET ns_additional = REPLACE(ns_additional,'wgCosmosRailBlacklistedNamespaces', 'wgCosmosRailDisabledNamespaces'); on mhglobal and testglobal [17:28:21] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.326 second response time [17:28:29] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.392 second response time [17:28:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:28:36] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.331 second response time [17:28:45] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 6.281 second response time [17:28:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:29:09] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.581 second response time [17:29:15] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [17:29:19] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 4s [17:29:38] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28480 bytes in 6.085 second response time [17:30:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:30:12] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.373 second response time [17:30:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:31:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.90, 3.73, 3.64 [17:31:20] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (START) [17:31:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:33:01] [02miraheze/ManageWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.1 [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/commit/7cf5d73d8db7 [17:33:03] [02miraheze/ManageWiki] 07dependabot[bot] 037cf5d73 - Bump stylelint-config-wikimedia from 0.13.0 to 0.13.1 [17:33:04] [02ManageWiki] 07dependabot[bot] opened pull request 03#364: Bump stylelint-config-wikimedia from 0.13.0 to 0.13.1 - 13https://github.com/miraheze/ManageWiki/pull/364 [17:33:04] [url] Page not found · GitHub · GitHub | github.com [17:33:05] [02ManageWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.1 - 13https://github.com/miraheze/ManageWiki [17:33:06] [url] Page not found · GitHub · GitHub | github.com [17:33:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.89, 3.38, 3.52 [17:33:07] [02ManageWiki] 07dependabot[bot] labeled pull request 03#364: Bump stylelint-config-wikimedia from 0.13.0 to 0.13.1 - 13https://github.com/miraheze/ManageWiki/pull/364 [17:33:07] [url] Page not found · GitHub · GitHub | github.com [17:33:09] [02ManageWiki] 07dependabot[bot] labeled pull request 03#364: Bump stylelint-config-wikimedia from 0.13.0 to 0.13.1 - 13https://github.com/miraheze/ManageWiki/pull/364 [17:33:09] [url] Page not found · GitHub · GitHub | github.com [17:33:10] [02miraheze/ManageWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/commit/3b02a9b2e3e6 [17:33:12] [02miraheze/ManageWiki] 07dependabot[bot] 033b02a9b - Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 [17:33:13] [02ManageWiki] 07dependabot[bot] opened pull request 03#365: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/ManageWiki/pull/365 [17:33:14] ... [17:33:15] [02ManageWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 - 13https://github.com/miraheze/ManageWiki [17:33:15] ... [17:33:16] [02ManageWiki] 07dependabot[bot] labeled pull request 03#365: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/ManageWiki/pull/365 [17:33:18] [02ManageWiki] 07dependabot[bot] labeled pull request 03#365: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/ManageWiki/pull/365 [17:33:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.52, 10.17, 9.60 [17:35:36] miraheze/ManageWiki - dependabot[bot] the build has errored. [17:36:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.11, 1.91, 1.93 [17:36:22] miraheze/ManageWiki - dependabot[bot] the build passed. [17:36:54] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/3b02a9b2e3e6...4a8b35da2740 [17:36:54] [url] Comparing 3b02a9b2e3e6...4a8b35da2740 · miraheze/ManageWiki · GitHub | github.com [17:36:55] [02miraheze/ManageWiki] 07Universal-Omega 034a8b35d - Update .stylelintrc.json [17:36:57] [02ManageWiki] 07Universal-Omega synchronize pull request 03#365: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/ManageWiki/pull/365 [17:36:57] ... [17:37:03] [02ManageWiki] 07Universal-Omega closed pull request 03#364: Bump stylelint-config-wikimedia from 0.13.0 to 0.13.1 - 13https://github.com/miraheze/ManageWiki/pull/364 [17:37:03] [url] Page not found · GitHub · GitHub | github.com [17:37:05] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/compare/eb7cb95641f6...7ae6a336b435 [17:37:05] [url] Comparing eb7cb95641f6...7ae6a336b435 · miraheze/ManageWiki · GitHub | github.com [17:37:06] [02miraheze/ManageWiki] 07dependabot[bot] 037ae6a33 - Bump stylelint-config-wikimedia from 0.13.0 to 0.13.1 (#364) [17:37:08] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.1 [17:37:09] [02ManageWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.1 - 13https://github.com/miraheze/ManageWiki [17:37:09] [url] Page not found · GitHub · GitHub | github.com [17:38:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.64, 1.82, 1.89 [17:38:58] [02puppet] 07paladox closed pull request 03#2714: mariadb: add grants for parsercache db - 13https://github.com/miraheze/puppet/pull/2714 [17:38:58] [url] Page not found · GitHub · GitHub | github.com [17:38:59] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9 [17:39:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8697ab832923...dc07c533bc2f [17:39:01] [url] Comparing 8697ab832923...dc07c533bc2f · miraheze/puppet · GitHub | github.com [17:39:02] [02miraheze/puppet] 07paladox 03dc07c53 - mariadb: add grants for parsercache db (#2714) [17:39:04] [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [17:39:04] [url] Page not found · GitHub · GitHub | github.com [17:39:50] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.53, 10.38, 9.54 [17:40:03] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.31, 1.94, 1.93 [17:40:26] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.71, 3.01, 2.50 [17:40:43] miraheze/ManageWiki - Universal-Omega the build passed. [17:40:59] [02ManageWiki] 07Universal-Omega closed pull request 03#365: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/ManageWiki/pull/365 [17:40:59] [url] Page not found · GitHub · GitHub | github.com [17:41:00] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/ManageWiki/compare/7ae6a336b435...870ab9ab204c [17:41:01] [url] Comparing 7ae6a336b435...870ab9ab204c · miraheze/ManageWiki · GitHub | github.com [17:41:02] [02miraheze/ManageWiki] 07dependabot[bot] 03870ab9a - Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 (#365) [17:41:03] [02ManageWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 - 13https://github.com/miraheze/ManageWiki [17:41:03] ... [17:41:05] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 [17:41:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.49, 11.68, 10.50 [17:41:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.61, 9.86, 8.93 [17:41:44] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.46, 11.28, 9.98 [17:42:31] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.656 second response time [17:43:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.79, 11.30, 10.13 [17:43:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.30, 10.40, 9.40 [17:44:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.83, 1.88 [17:44:20] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 3.09, 3.14, 2.68 [17:44:33] miraheze/ManageWiki - Universal-Omega the build passed. [17:45:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.06, 11.49, 10.72 [17:45:23] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.89, 9.99, 9.24 [17:45:32] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.14, 11.90, 10.48 [17:47:16] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.939 second response time [17:47:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.17, 12.26, 11.09 [17:47:26] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.81, 10.64, 10.18 [17:47:46] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.842 second response time [17:48:04] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.40, 1.93, 1.89 [17:48:23] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 7.831 second response time [17:48:25] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.380 second response time [17:49:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.41, 3.98, 3.70 [17:49:11] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:51:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.04, 3.50, 3.56 [17:51:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.64, 10.05, 10.05 [17:51:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.36, 11.77, 11.20 [17:51:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/80e4d4dd4381 [17:51:31] [02miraheze/puppet] 07paladox 0380e4d4d - Install prometheus131 [17:51:33] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [17:51:33] [url] Page not found · GitHub · GitHub | github.com [17:51:34] [02puppet] 07paladox opened pull request 03#2719: Install prometheus131 - 13https://github.com/miraheze/puppet/pull/2719 [17:51:35] ... [17:51:37] PROBLEM - docse.tk - LetsEncrypt on sslhost is CRITICAL: connect to address docse.tk and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [17:51:53] !log [@test101] starting deploy of {'config': True} to all [17:51:54] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [17:52:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:52:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:52:44] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:47] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:11] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.25, 10.09, 9.88 [17:53:14] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:50] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:54:25] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.92, 3.95, 3.70 [17:55:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.28, 12.19, 11.49 [17:57:10] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:57:12] [02puppet] 07paladox closed pull request 03#2719: Install prometheus131 - 13https://github.com/miraheze/puppet/pull/2719 [17:57:13] [url] Page not found · GitHub · GitHub | github.com [17:57:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dc07c533bc2f...79279ce0ef32 [17:57:14] [url] Comparing dc07c533bc2f...79279ce0ef32 · miraheze/puppet · GitHub | github.com [17:57:15] [02miraheze/puppet] 07paladox 0379279ce - Install prometheus131 (#2719) [17:57:17] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [17:57:18] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [17:57:19] [url] Page not found · GitHub · GitHub | github.com [17:57:22] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.02, 11.33, 11.26 [18:01:13] PROBLEM - test101 Puppet on test101 is WARNING: WARNING: Puppet is currently disabled, message: Out of sync, last run 2 minutes ago with 0 failures [18:03:55] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.855 second response time [18:04:00] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:04:06] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:04:07] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:04:55] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw111 [18:05:17] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:05:24] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:05:25] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.56, 1.67, 1.96 [18:05:27] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.517 second response time [18:05:45] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.146 second response time [18:05:54] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.053 second response time [18:05:57] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.313 second response time [18:06:52] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [18:07:13] PROBLEM - prometheus101 Prometheus on prometheus101 is CRITICAL: connect to address 2a10:6740::6:110 and port 9090: Connection refused [18:07:24] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.13, 1.93, 2.02 [18:07:24] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.80, 8.87, 10.14 [18:10:26] PROBLEM - docse.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for docse.tk could not be found [18:12:19] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:14] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.015 second response time [18:13:49] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:14:24] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.683 second response time [18:15:13] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [18:15:46] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [18:16:34] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.778 second response time [18:16:37] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw121 [18:16:48] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw121 [18:17:12] !log [universalomega@test101] starting deploy of {'pull': 'config', 'branch': 'paladox-patch-2', 'config': True} to all [18:17:16] !log [universalomega@test101] finished deploy of {'pull': 'config', 'branch': 'paladox-patch-2', 'config': True} to all - SUCCESS in 3s [18:17:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:18:37] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [18:18:46] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [18:19:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:19:43] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:31] RECOVERY - prometheus101 Prometheus on prometheus101 is OK: TCP OK - 0.004 second response time on 2a10:6740::6:110 port 9090 [18:20:37] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:40] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:45] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:01] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:01] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:08] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:16] PROBLEM - prometheus101 Current Load on prometheus101 is CRITICAL: CRITICAL - load average: 2.32, 1.59, 0.95 [18:21:46] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.726 second response time [18:22:51] !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (END - exit=0) [18:23:06] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:11] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:14] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.497 second response time [18:23:41] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:49] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/e4f0ffa6d178...3e7ad662cf9e [18:23:49] [url] Comparing e4f0ffa6d178...3e7ad662cf9e · miraheze/dns · GitHub | github.com [18:23:50] [02miraheze/dns] 07Universal-Omega 033e7ad66 - Add arcanewaterswiki.org zone (#307) [18:23:52] [02dns] 07Reception123 closed pull request 03#307: Add arcanewaterswiki.org zone - 13https://github.com/miraheze/dns/pull/307 [18:23:52] [url] Page not found · GitHub · GitHub | github.com [18:24:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:00] !log CREATE TABLE pc ( keyname VARBINARY(255) DEFAULT '' NOT NULL, value MEDIUMBLOB DEFAULT NULL, exptime BINARY(14) NOT NULL, modtoken VARCHAR(17) DEFAULT '00000000000000000' NOT NULL, flags INT UNSIGNED DEFAULT NULL, INDEX exptime (exptime), PRIMARY KEY(keyname) ); on parsercache database [18:25:07] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.837 second response time [18:25:11] PROBLEM - prometheus101 Current Load on prometheus101 is WARNING: WARNING - load average: 1.83, 1.86, 1.22 [18:25:12] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.283 second response time [18:25:41] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 1.551 second response time [18:25:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.08, 3.36, 3.92 [18:26:47] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.023 second response time [18:26:48] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.332 second response time [18:26:53] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:26:58] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:27:11] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 6.510 second response time [18:27:15] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.014 second response time [18:27:16] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.758 second response time [18:27:29] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:27:29] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:27:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.80, 9.06, 8.82 [18:28:28] PROBLEM - es131 Puppet on es131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:28:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.22, 4.00, 4.08 [18:28:43] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown): File[/etc/sysctl.d],File[/home/johnflewis],File[/home/paladox],File[/home/reception] [18:28:43] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:28:53] uh what? [18:32:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.46, 9.00, 8.83 [18:32:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.71, 3.25, 3.76 [18:32:46] RECOVERY - prometheus101 Current Load on prometheus101 is OK: OK - load average: 1.27, 1.57, 1.35 [18:32:50] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.850 second response time [18:32:50] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.582 second response time [18:34:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.69, 4.01, 3.96 [18:35:55] PROBLEM - prometheus101 Puppet on prometheus101 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 15 minutes ago with 0 failures [18:35:58] PROBLEM - prometheus101 Prometheus on prometheus101 is CRITICAL: connect to address 2a10:6740::6:110 and port 9090: Connection refused [18:36:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.59, 3.47, 3.78 [18:37:08] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:13] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:38:19] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.00, 2.02, 1.17 [18:38:36] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.452 second response time [18:38:54] Reception123: it'll be puppet OOM'd [18:40:14] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.47, 1.47, 1.07 [18:40:59] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:40:59] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:41:00] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:41:00] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:41:03] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:41:03] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:57] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.078 second response time [18:42:58] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.684 second response time [18:42:59] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:59] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 5.952 second response time [18:42:59] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:43:03] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.367 second response time [18:43:13] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.175 second response time [18:43:26] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.551 second response time [18:44:25] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.48, 3.97, 3.87 [18:44:34] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.665 second response time [18:44:45] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.196 second response time [18:44:45] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.719 second response time [18:44:58] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.020 second response time [18:45:00] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 0.491 second response time [18:45:02] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.715 second response time [18:45:02] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.318 second response time [18:45:05] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.335 second response time [18:45:14] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.251 second response time [18:45:32] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.323 second response time [18:45:33] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 3.076 second response time [18:49:20] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:25] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:31] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:46] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:52] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:56] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:51:02] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:51:05] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:51:08] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:51:32] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:51:33] [02miraheze/TSPortal] 07OwenBaines pushed 034 commits to 03master [+0/-0/±11] 13https://github.com/miraheze/TSPortal/compare/6b6eb1049b08...2ce35447995c [18:51:34] [url] Comparing 6b6eb1049b08...2ce35447995c · miraheze/TSPortal · GitHub | github.com [18:51:35] [02miraheze/TSPortal] 07OwenBaines 03cb7e9b0 - Fix models passed through to events [18:51:36] [02miraheze/TSPortal] 07OwenBaines 0333e1ab9 - Upgrade numerous packages [18:51:38] [02miraheze/TSPortal] 07OwenBaines 039f74e2a - User assigned needs to be passed ID not object [18:51:39] [02miraheze/TSPortal] 07OwenBaines 032ce3544 - [Release] Version 6 [18:51:41] [02miraheze/TSPortal] 07OwenBaines tagged 032ce3544 as 03v6 13https://github.com/miraheze/TSPortal/commit/2ce35447995c4cef6fd9521e8813754f3bd3bb5b [18:51:42] [02TSPortal] 07OwenBaines created tag 03v6 - 13https://github.com/miraheze/TSPortal [18:51:43] [url] Page not found · GitHub · GitHub | github.com [18:51:59] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.626 second response time [18:52:03] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28488 bytes in 8.711 second response time [18:52:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.77, 3.74, 3.85 [18:54:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.76, 4.02, 3.95 [18:55:19] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:55:21] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:55:33] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:55:36] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:55:40] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:56:06] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:56:19] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:56:25] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:56:27] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:56:33] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:56:53] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:56:54] RECOVERY - es131 Puppet on es131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:57:14] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.319 second response time [18:57:22] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:57:27] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 9.913 second response time [18:57:35] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.019 second response time [18:57:38] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.314 second response time [18:58:07] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 1.835 second response time [18:58:22] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.061 second response time [18:58:25] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 9.536 second response time [18:58:32] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 6.979 second response time [18:58:40] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 27673 bytes in 1.260 second response time [18:58:55] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.563 second response time [18:58:56] !log [universalomega@test101] starting deploy of {'pull': 'config', 'branch': 'paladox-patch-2', 'config': True, 'ignoretime': True} to all [18:58:59] !log [universalomega@test101] finished deploy of {'pull': 'config', 'branch': 'paladox-patch-2', 'config': True, 'ignoretime': True} to all - SUCCESS in 2s [19:00:03] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.321 second response time [19:00:17] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:00:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:00:41] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 32405 bytes in 1.669 second response time [19:01:03] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.21, 9.96, 9.43 [19:01:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:01:32] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.637 second response time [19:02:19] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.170 second response time [19:02:46] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:02:55] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:03:02] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.19, 9.74, 9.42 [19:04:41] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:43] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:50] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:28] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:55] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:14] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.25, 1.83, 1.43 [19:08:28] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw112 [19:09:14] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk[Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk [19:10:00] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.798 second response time [19:10:03] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.796 second response time [19:10:14] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.15, 1.60, 1.39 [19:10:28] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [19:10:51] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:11:55] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.576 second response time [19:12:12] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.685 second response time [19:12:25] [02mw-config] 07Universal-Omega reviewed pull request 03#4829 commit - 13https://github.com/miraheze/mw-config/pull/4829#discussion_r934834410 [19:12:25] [url] cache: support using memcache for parsoid cache by paladox · Pull Request #4829 · miraheze/mw-config · GitHub | github.com [19:12:36] paladox: ^ [19:12:44] [02mw-config] 07paladox synchronize pull request 03#4829: cache: support using memcache for parsoid cache - 13https://github.com/miraheze/mw-config/pull/4829 [19:12:45] [url] Page not found · GitHub · GitHub | github.com [19:12:46] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/fc6c49827a9c...a1beceb70e65 [19:12:46] [url] Comparing fc6c49827a9c...a1beceb70e65 · miraheze/mw-config · GitHub | github.com [19:12:47] [02miraheze/mw-config] 07paladox 03a1beceb - Update GlobalCache.php [19:12:55] CosmicAlpha: you can edit it if you want [19:13:54] miraheze/mw-config - paladox the build passed. [19:14:13] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.52, 2.30, 1.52 [19:14:21] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:24] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:56] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.022 second response time [19:15:06] paladox: yeah true, but with that, testing works. I tested by commenting out memcached in multiwrite, to ensure the fallback DB inserts, with that it correctly inserts now into `parsercache.pc`, and use memcached by default also. That is however only tested on test101/betawiki do not know if any other tests need done. [19:15:14] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.292 second response time [19:15:25] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 6.189 second response time [19:15:25] !log [universalomega@test101] starting deploy of {'pull': 'config', 'branch': 'paladox-patch-2', 'config': True, 'ignoretime': True} to all [19:15:28] nice [19:15:30] !log [universalomega@test101] finished deploy of {'pull': 'config', 'branch': 'paladox-patch-2', 'config': True, 'ignoretime': True} to all - SUCCESS in 4s [19:16:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:16:13] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.27, 1.82, 1.44 [19:17:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:17:32] paladox: want me to deploy, and monitor load/perf? I can't get good perf report on test101, since no traffic it would be decent regardless, so should I deploy to prod, and monitor it? [19:17:39] yup! [19:18:13] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.29, 3.06, 1.99 [19:19:52] also you won't get a good perf reading on test101 [19:19:56] it only has one cpu [19:20:15] [02mw-config] 07Universal-Omega closed pull request 03#4829: cache: support using memcache for parsoid cache - 13https://github.com/miraheze/mw-config/pull/4829 [19:20:16] [url] Page not found · GitHub · GitHub | github.com [19:20:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/139e8c207b47...5f8fc17e2648 [19:20:17] [url] Comparing 139e8c207b47...5f8fc17e2648 · miraheze/mw-config · GitHub | github.com [19:20:18] [02miraheze/mw-config] 07paladox 035f8fc17 - cache: support using memcache for parsoid cache (#4829) [19:20:20] [02miraheze/mw-config] 07Universal-Omega deleted branch 03paladox-patch-2 [19:20:21] [02mw-config] 07Universal-Omega deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/mw-config [19:20:22] [url] Page not found · GitHub · GitHub | github.com [19:20:44] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [19:20:47] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 2s [19:21:03] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [19:21:39] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.217 second response time [19:21:50] miraheze/mw-config - Universal-Omega the build passed. [19:22:17] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:20] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.217 second response time [19:22:25] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:28] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:34] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 91s [19:22:38] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.304 second response time [19:22:39] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.020 second response time [19:22:44] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:47] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:49] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:50] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:14] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.66, 2.89, 2.15 [19:23:34] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.903 second response time [19:23:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:23:47] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:53] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:24:05] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:24:13] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.24, 1.45, 1.63 [19:24:15] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:24:28] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [19:25:17] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:25:55] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:26:28] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [19:26:29] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:26:43] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:14] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.96, 1.72, 1.83 [19:27:57] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:58] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.589 second response time [19:28:01] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.328 second response time [19:28:05] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.159 second response time [19:28:09] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:19] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [19:28:19] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 5.096 second response time [19:28:22] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 1.005 second response time [19:28:34] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.34, 9.90, 8.72 [19:28:35] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.223 second response time [19:28:47] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 6.247 second response time [19:28:48] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.631 second response time [19:29:02] Saw something in the scrollback about an SRE member getting OOM'ed, and wondered if that's related to the extreme slowness and 502s I've been getting most of today on `commonswiki`, `metawiki`, and `testwiki` [19:29:14] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.78, 1.39, 1.69 [19:29:23] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.386 second response time [19:30:32] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.87, 9.71, 8.79 [19:31:58] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.64, 4.00 [19:32:12] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.238 second response time [19:32:28] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:28] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:32] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:46] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:58] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:02] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:10] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:10] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 3.336 second response time [19:33:15] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:16] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:18] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.782 second response time [19:33:19] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.976 second response time [19:33:24] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.039 second response time [19:33:44] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:02] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.314 second response time [19:34:11] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.80, 4.00, 4.08 [19:34:13] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.019 second response time [19:34:24] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.01, 9.87, 9.00 [19:34:33] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.330 second response time [19:34:34] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:52] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:36:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.92, 3.62, 3.93 [19:36:19] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.814 second response time [19:36:22] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.83, 9.76, 9.06 [19:36:30] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.266 second response time [19:36:42] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 4.747 second response time [19:36:56] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 3.543 second response time [19:37:07] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.477 second response time [19:37:37] dmehus: the slowness has been going on for a good few weeks [19:37:51] But no puppetserver OOM'ing likely doesn't affect it [19:38:55] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.337 second response time [19:39:08] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 9.747 second response time [19:39:13] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 8.765 second response time [19:39:23] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.502 second response time [19:39:33] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.021 second response time [19:39:40] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.661 second response time [19:39:55] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.044 second response time [19:40:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.35, 10.58, 9.73 [19:42:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/0ff39f41251b [19:42:10] [02miraheze/puppet] 07paladox 030ff39f4 - Remove prometheus101 [19:42:11] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.20, 3.82, 3.89 [19:42:12] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [19:42:12] [url] Page not found · GitHub · GitHub | github.com [19:42:13] [02puppet] 07paladox opened pull request 03#2720: Remove prometheus101 - 13https://github.com/miraheze/puppet/pull/2720 [19:42:14] ... [19:42:28] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.21, 10.11, 9.67 [19:42:31] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/3c0596fd4c64 [19:42:33] [02miraheze/dns] 07paladox 033c0596f - Remove prometheus101 from dns [19:42:34] [02dns] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/dns [19:42:34] [url] Page not found · GitHub · GitHub | github.com [19:42:36] [02dns] 07paladox opened pull request 03#312: Remove prometheus101 from dns - 13https://github.com/miraheze/dns/pull/312 [19:42:36] ... [19:42:38] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.713 second response time [19:42:42] !log shutdown and remove vm prometheus101 [19:42:47] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.774 second response time [19:42:48] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.976 second response time [19:42:59] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:43:21] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:43:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:43:33] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.04, 10.08, 9.36 [19:44:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.99, 3.69, 3.83 [19:44:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.14, 10.58, 9.67 [19:45:32] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 10.14, 10.10, 9.46 [19:46:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.22, 10.64, 9.98 [19:47:50] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:02] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:11] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.70, 4.01, 3.92 [19:48:18] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.83, 9.84, 9.76 [19:48:23] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.80, 9.98, 9.65 [19:49:00] RhinosF1, ah, any idea on cause of the recent slowness? I know CosmicAlpha's reconfiguration a couple months ago really helped things [19:49:06] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:49:13] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.11, 1.58, 1.38 [19:49:14] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk[Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk [19:49:21] dmehus: Reception123 just asked me the same question [19:49:24] And no I do not [19:49:45] it could be connectivity, but it is quite odd [19:50:23] RhinosF1, Reception123: ah [19:51:13] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.34, 1.60, 1.42 [19:51:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.32, 10.46, 9.84 [19:51:51] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 8.259 second response time [19:52:00] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.903 second response time [19:52:15] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 9.143 second response time [19:53:26] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.44, 9.44, 9.58 [19:54:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.42, 3.86, 3.91 [19:54:55] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:02] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:13] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:29] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.248 second response time [19:55:29] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:31] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:47] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:48] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:49] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:52] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:54] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:56] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:06] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.009 second response time [19:56:06] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.243 second response time [19:56:14] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:15] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:18] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:26] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:35] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:42] PROBLEM - prometheus101 Puppet on prometheus101 is CRITICAL: connect to address 2a10:6740::6:110 port 5666: No route to hostconnect to host 2a10:6740::6:110 port 5666: No route to host [19:56:44] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:47] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:29] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 2.210 second response time [19:57:31] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.335 second response time [19:57:42] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw111 [19:57:44] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.437 second response time [19:57:45] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [19:57:49] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [19:57:51] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.330 second response time [19:58:09] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.377 second response time [19:58:23] PROBLEM - ping6 on prometheus101 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:110) [19:58:28] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw112 mw121 [19:58:30] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.024 second response time [19:58:33] PROBLEM - prometheus101 ferm_active on prometheus101 is CRITICAL: connect to address 2a10:6740::6:110 port 5666: No route to hostconnect to host 2a10:6740::6:110 port 5666: No route to host [19:58:49] PROBLEM - prometheus101 Current Load on prometheus101 is CRITICAL: connect to address 2a10:6740::6:110 port 5666: No route to hostconnect to host 2a10:6740::6:110 port 5666: No route to host [19:58:57] PROBLEM - prometheus101 Disk Space on prometheus101 is CRITICAL: connect to address 2a10:6740::6:110 port 5666: No route to hostconnect to host 2a10:6740::6:110 port 5666: No route to host [19:59:00] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.990 second response time [19:59:08] PROBLEM - prometheus101 PowerDNS Recursor on prometheus101 is CRITICAL: connect to address 2a10:6740::6:110 port 5666: No route to hostconnect to host 2a10:6740::6:110 port 5666: No route to host [19:59:08] PROBLEM - prometheus101 NTP time on prometheus101 is CRITICAL: connect to address 2a10:6740::6:110 port 5666: No route to hostconnect to host 2a10:6740::6:110 port 5666: No route to host [19:59:14] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [19:59:18] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.309 second response time [19:59:25] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 3.762 second response time [19:59:31] PROBLEM - prometheus101 SSH on prometheus101 is CRITICAL: connect to address 2a10:6740::6:110 and port 22: No route to host [19:59:31] PROBLEM - prometheus101 conntrack_table_size on prometheus101 is CRITICAL: connect to address 2a10:6740::6:110 port 5666: No route to hostconnect to host 2a10:6740::6:110 port 5666: No route to host [19:59:39] PROBLEM - Host prometheus101 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:110) [19:59:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/79279ce0ef32...737aa8f54c09 [19:59:42] [url] Comparing 79279ce0ef32...737aa8f54c09 · miraheze/puppet · GitHub | github.com [19:59:43] [02miraheze/puppet] 07paladox 03737aa8f - Remove prometheus101 (#2720) [19:59:44] [02puppet] 07paladox closed pull request 03#2720: Remove prometheus101 - 13https://github.com/miraheze/puppet/pull/2720 [19:59:45] [url] Page not found · GitHub · GitHub | github.com [19:59:46] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [19:59:47] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [19:59:48] [url] Page not found · GitHub · GitHub | github.com [19:59:52] [02dns] 07paladox closed pull request 03#312: Remove prometheus101 from dns - 13https://github.com/miraheze/dns/pull/312 [19:59:52] ... [19:59:53] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/3e7ad662cf9e...1c30f131ef3d [19:59:54] [url] Comparing 3e7ad662cf9e...1c30f131ef3d · miraheze/dns · GitHub | github.com [19:59:55] [02miraheze/dns] 07paladox 031c30f13 - Remove prometheus101 from dns (#312) [20:00:07] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 4.477 second response time [20:00:08] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.952 second response time [20:00:08] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.064 second response time [20:00:28] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [20:01:42] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [20:01:46] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 2 backends are down. mw101 mw102 [20:02:12] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:02:14] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 4.450 second response time [20:02:18] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.924 second response time [20:02:23] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.414 second response time [20:02:27] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 1.740 second response time [20:02:28] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:02:28] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:02:35] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:02:37] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.219 second response time [20:02:49] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 9.692 second response time [20:03:01] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.024 second response time [20:03:01] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.307 second response time [20:03:18] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:03:26] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 5.271 second response time [20:03:32] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.307 second response time [20:03:40] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:03:46] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [20:03:52] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.030 second response time [20:04:00] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.332 second response time [20:04:04] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.875 second response time [20:04:11] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.022 second response time [20:04:17] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 3.212 second response time [20:04:23] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.657 second response time [20:04:27] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.337 second response time [20:04:30] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:04:33] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:04:34] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.031 second response time [20:04:36] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:05:08] PROBLEM - bast101 Puppet on bast101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:06:11] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.78, 4.23, 3.88 [20:06:12] PROBLEM - mw111 Puppet on mw111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:06:18] PROBLEM - mon111 Puppet on mon111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:07:03] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:10:03] RECOVERY - mon111 Puppet on mon111 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [20:10:33] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.15, 1.46, 1.85 [20:14:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.25, 10.78, 9.38 [20:14:27] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.00, 1.26, 1.68 [20:16:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.11, 10.14, 9.31 [20:21:58] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.18, 10.82, 9.86 [20:22:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.59, 10.32, 8.90 [20:22:29] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.032 second response time [20:22:34] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.411 second response time [20:23:01] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.098 second response time [20:23:04] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.407 second response time [20:23:10] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw111 [20:23:20] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.350 second response time [20:23:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.71, 9.13, 9.35 [20:24:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.31, 10.45, 9.11 [20:25:05] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 14 backends are healthy [20:25:47] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.57, 2.25, 1.51 [20:27:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.33, 1.85, 1.45 [20:29:29] [02mediawiki] 07Universal-Omega closed pull request 03#8389: Bump skins/Cosmos from `4fe5e5a` to `d1163b5` - 13https://github.com/miraheze/mediawiki/pull/8389 [20:29:29] [url] Page not found · GitHub · GitHub | github.com [20:29:30] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/skins/Cosmos-d1163b5 [20:29:32] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/skins/Cosmos-d1163b5 - 13https://github.com/miraheze/mediawiki [20:29:32] [url] Page not found · GitHub · GitHub | github.com [20:29:33] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/c9097b8905f6...c4268da08dba [20:29:34] [url] Comparing c9097b8905f6...c4268da08dba · miraheze/mediawiki · GitHub | github.com [20:29:35] [02miraheze/mediawiki] 07dependabot[bot] 03c4268da - Bump skins/Cosmos from `4fe5e5a` to `d1163b5` (#8389) [20:29:38] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:29:44] [02mediawiki] 07Universal-Omega closed pull request 03#8383: Bump extensions/SimpleBlogPage from `3173559` to `9f7e016` - 13https://github.com/miraheze/mediawiki/pull/8383 [20:29:44] [url] Page not found · GitHub · GitHub | github.com [20:29:45] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/c4268da08dba...fb1e8a1be38a [20:29:46] [url] Comparing c4268da08dba...fb1e8a1be38a · miraheze/mediawiki · GitHub | github.com [20:29:46] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.79, 1.50, 1.37 [20:29:47] [02miraheze/mediawiki] 07dependabot[bot] 03fb1e8a1 - Bump extensions/SimpleBlogPage from `3173559` to `9f7e016` (#8383) [20:29:48] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/extensions/SimpleBlogPage-9f7e016 [20:29:50] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_38/extensions/SimpleBlogPage-9f7e016 - 13https://github.com/miraheze/mediawiki [20:29:50] [url] Page not found · GitHub · GitHub | github.com [20:30:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.13, 9.31, 9.14 [20:30:39] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk [20:31:08] !log [universalomega@mwtask111] starting deploy of {'world': True} to all [20:31:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:31:18] !log [universalomega@test101] starting deploy of {'world': True} to all [20:31:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:31:38] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.914 second response time [20:33:03] RECOVERY - bast101 Puppet on bast101 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [20:35:01] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [20:35:19] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.78, 10.76, 9.61 [20:37:01] RECOVERY - mw111 Puppet on mw111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:37:09] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/737aa8f54c09...d06753bf9cc0 [20:37:09] [url] Comparing 737aa8f54c09...d06753bf9cc0 · miraheze/puppet · GitHub | github.com [20:37:10] [02miraheze/puppet] 07JohnFLewis 03d06753b - reports: upgrade to v6 [20:37:17] !log [universalomega@test101] finished deploy of {'world': True} to all - SUCCESS in 358s [20:37:17] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.76, 10.67, 9.71 [20:37:36] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.42, 10.07, 9.09 [20:37:42] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-3 [20:37:43] [02dns] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/dns [20:37:43] [url] Page not found · GitHub · GitHub | github.com [20:37:48] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:37:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:38:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.92, 1.71 [20:38:33] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:38:58] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/9c7ba4a49865 [20:39:00] [02miraheze/dns] 07paladox 039c7ba4a - Add mw131 and 132 to dns [20:39:01] [02dns] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/dns [20:39:02] ... [20:39:03] [02dns] 07paladox opened pull request 03#313: Add mw131 and 132 to dns - 13https://github.com/miraheze/dns/pull/313 [20:39:03] [url] Page not found · GitHub · GitHub | github.com [20:39:07] [02dns] 07paladox closed pull request 03#313: Add mw131 and 132 to dns - 13https://github.com/miraheze/dns/pull/313 [20:39:08] [url] Page not found · GitHub · GitHub | github.com [20:39:09] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/1c30f131ef3d...03525afeb797 [20:39:09] [url] Comparing 1c30f131ef3d...03525afeb797 · miraheze/dns · GitHub | github.com [20:39:10] [02miraheze/dns] 07paladox 0303525af - Add mw131 and 132 to dns (#313) [20:39:12] [02dns] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/dns [20:39:12] [url] Page not found · GitHub · GitHub | github.com [20:39:13] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-3 [20:39:35] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.10, 9.84, 9.13 [20:39:38] !log [universalomega@mwtask111] finished deploy of {'world': True} to all - SUCCESS in 510s [20:39:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:40:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.54, 10.58, 9.88 [20:40:26] [02puppet] 07RhinosF1 opened pull request 03#2721: mwdeploy: add new mw13 servers - 13https://github.com/miraheze/puppet/pull/2721 [20:40:26] ... [20:41:13] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.45, 10.01, 9.71 [20:41:57] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.09, 1.51, 1.59 [20:42:00] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.00, 8.56, 9.23 [20:42:11] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.011 second response time [20:42:42] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:42:55] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:03] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:05] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:05] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:12] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:16] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:16] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:19] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:28] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:38] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:38] [02puppet] 07RhinosF1 opened pull request 03#2722: mwdeploy: swap test101 for test131 - 13https://github.com/miraheze/puppet/pull/2722 [20:43:39] [url] Page not found · GitHub · GitHub | github.com [20:43:39] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:39] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:55] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:56] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:58] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw122 [20:44:01] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:03] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:05] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:05] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:07] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:07] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:45:15] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 4.626 second response time [20:45:18] [02miraheze/MirahezeDebug] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MirahezeDebug/commit/9213c81459ea [20:45:19] [02miraheze/MirahezeDebug] 07Universal-Omega 039213c81 - Update servers [20:45:21] [02MirahezeDebug] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/MirahezeDebug [20:45:21] ... [20:45:22] [02MirahezeDebug] 07Universal-Omega opened pull request 03#7: Update servers - 13https://github.com/miraheze/MirahezeDebug/pull/7 [20:45:23] [url] Page not found · GitHub · GitHub | github.com [20:45:41] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.948 second response time [20:45:44] [02miraheze/MirahezeDebug] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MirahezeDebug/compare/9213c81459ea...6a224c0616b5 [20:45:45] [url] Comparing 9213c81459ea...6a224c0616b5 · miraheze/MirahezeDebug · GitHub | github.com [20:45:46] [02miraheze/MirahezeDebug] 07Universal-Omega 036a224c0 - Update background.js [20:45:47] [02MirahezeDebug] 07Universal-Omega synchronize pull request 03#7: Update servers - 13https://github.com/miraheze/MirahezeDebug/pull/7 [20:45:48] [url] Page not found · GitHub · GitHub | github.com [20:46:00] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.455 second response time [20:46:10] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.619 second response time [20:46:11] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.258 second response time [20:47:21] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.023 second response time [20:47:21] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.027 second response time [20:47:50] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:47:54] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [20:48:12] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 7.648 second response time [20:48:13] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.038 second response time [20:48:44] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 11.32, 7.95, 5.29 [20:50:08] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:12] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.241 second response time [20:50:30] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:37] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:40] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:44] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:46] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.969 second response time [20:51:49] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:50] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:52:00] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.045 second response time [20:52:06] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:52:17] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 9.794 second response time [20:52:30] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.364 second response time [20:52:39] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.798 second response time [20:52:40] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.660 second response time [20:52:44] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.60, 6.77, 5.53 [20:53:06] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 1.899 second response time [20:53:16] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:53:18] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.510 second response time [20:53:40] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.312 second response time [20:54:00] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.816 second response time [20:54:14] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 7.691 second response time [20:54:14] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 7.235 second response time [20:54:33] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.204 second response time [20:55:18] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.023 second response time [20:55:58] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 8.857 second response time [20:56:00] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 2.822 second response time [20:56:00] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.143 second response time [20:56:14] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 8.450 second response time [20:56:22] PROBLEM - mw112 NTP time on mw112 is WARNING: NTP WARNING: Offset -0.1052689552 secs [20:56:26] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.871 second response time [20:57:57] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.328 second response time [20:57:58] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.052 second response time [21:06:32] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 2.172 second response time [21:06:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.66, 10.23, 9.25 [21:06:40] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 2.020 second response time [21:06:49] PROBLEM - wiki.nevillepedia.eu - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:07:14] PROBLEM - wikimoma.art - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:07:16] PROBLEM - taswin.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:07:45] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 6.346 second response time [21:07:56] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [21:08:14] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:08:26] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 9.149 second response time [21:08:50] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 3 backends are down. mw102 mw112 mw121 [21:09:14] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [21:09:56] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is WARNING: WARNING - NGINX Error Rate is 45% [21:09:56] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:10:00] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/4fab18defa9a [21:10:02] [02miraheze/ssl] 07Universal-Omega 034fab18d - Add elsterclopedie.marithaime.nl cert [21:10:03] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [21:10:04] [url] Page not found · GitHub · GitHub | github.com [21:10:05] [02ssl] 07Universal-Omega opened pull request 03#581: Add elsterclopedie.marithaime.nl cert - 13https://github.com/miraheze/ssl/pull/581 [21:10:05] ... [21:10:20] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:10:28] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.96, 10.10, 9.38 [21:10:29] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.19, 1.74, 1.29 [21:11:07] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4fab18defa9a...ced53a7f18b7 [21:11:08] [url] Comparing 4fab18defa9a...ced53a7f18b7 · miraheze/ssl · GitHub | github.com [21:11:09] [02miraheze/ssl] 07Universal-Omega 03ced53a7 - Update certs.yaml [21:11:10] [02ssl] 07Universal-Omega synchronize pull request 03#581: Add elsterclopedie.marithaime.nl cert - 13https://github.com/miraheze/ssl/pull/581 [21:11:11] [url] Page not found · GitHub · GitHub | github.com [21:12:17] PROBLEM - m.miraheze.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:18] PROBLEM - www.zetapedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.zetapedia.com' expires in 15 day(s) (Wed 17 Aug 2022 21:08:43 GMT +0000). [21:12:28] PROBLEM - m33ks.ml - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:28] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.05, 1.39, 1.21 [21:12:34] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:49] PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:13:43] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:14:12] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:14:13] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Tue 06 Sep 2022 21:18:17 GMT +0000. [21:14:29] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3406 bytes in 0.017 second response time [21:14:43] RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 6% [21:14:58] [02ssl] 07Universal-Omega closed pull request 03#581: Add elsterclopedie.marithaime.nl cert - 13https://github.com/miraheze/ssl/pull/581 [21:14:58] [url] Page not found · GitHub · GitHub | github.com [21:14:59] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:15:01] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/7955c7f70c75...d8b57c3699e4 [21:15:01] [url] Comparing 7955c7f70c75...d8b57c3699e4 · miraheze/ssl · GitHub | github.com [21:15:02] [02miraheze/ssl] 07Universal-Omega 03d8b57c3 - Add elsterclopedie.marithaime.nl cert (#581) [21:15:04] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [21:15:04] [url] Page not found · GitHub · GitHub | github.com [21:15:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.13, 3.27, 3.91 [21:15:46] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: HTTP CRITICAL - No data received from host [21:16:12] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:17:26] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw111 [21:17:42] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 0.322 second response time [21:18:17] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:18:19] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 6.290 second response time [21:19:05] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:19:10] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:19:22] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:20:07] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:21:03] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 4.034 second response time [21:21:07] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 2.084 second response time [21:21:10] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.69, 3.86, 3.92 [21:21:23] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [21:21:24] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 5.552 second response time [21:21:39] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.018 second response time [21:21:39] !log reset reports.mh.o database [21:21:51] !log added agent@ to trustandsafety ldap group [21:21:59] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:22:01] !log added universalomega@ to board ldap group [21:22:06] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [21:22:21] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.309 second response time [21:22:28] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 1.050 second response time [21:22:32] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.481 second response time [21:22:34] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:23:07] [02mw-config] 07Universal-Omega closed pull request 03#4736: Set new RemovePII DPA validation variables - 13https://github.com/miraheze/mw-config/pull/4736 [21:23:08] [url] Page not found · GitHub · GitHub | github.com [21:23:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5f8fc17e2648...f026df0e366f [21:23:10] [url] Comparing 5f8fc17e2648...f026df0e366f · miraheze/mw-config · GitHub | github.com [21:23:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.09, 3.53, 3.79 [21:23:10] [02miraheze/mw-config] 07Universal-Omega 03f026df0 - Set new RemovePII DPA validation variables (#4736) [21:23:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:23:12] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:23:13] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [21:23:14] [url] Page not found · GitHub · GitHub | github.com [21:23:42] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [21:23:48] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [21:24:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:24:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:24:25] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [21:24:30] miraheze/mw-config - Universal-Omega the build passed. [21:24:37] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 54s [21:24:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:25:17] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all [21:25:18] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to all - SUCCESS in 1s [21:26:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:26:09] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:26:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:27:10] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.86, 3.94, 3.87 [21:28:12] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 5.784 second response time [21:28:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:31:20] PROBLEM - mw131 conntrack_table_size on mw131 is UNKNOWN: NRPE: Unable to read output [21:31:24] PROBLEM - mw131 ferm_active on mw131 is UNKNOWN: NRPE: Unable to read output [21:31:44] PROBLEM - mw132 Puppet on mw132 is UNKNOWN: NRPE: Unable to read output [21:31:55] PROBLEM - mw131 Puppet on mw131 is UNKNOWN: NRPE: Unable to read output [21:32:00] PROBLEM - mw132 conntrack_table_size on mw132 is UNKNOWN: NRPE: Unable to read output [21:32:04] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:32:04] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:32:06] PROBLEM - mw132 ferm_active on mw132 is UNKNOWN: NRPE: Unable to read output [21:33:03] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:34:07] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 8.538 second response time [21:34:08] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 6.284 second response time [21:35:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.71, 3.48, 3.63 [21:35:08] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 7.927 second response time [21:35:37] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 8.236 second response time [21:35:39] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk [21:35:39] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [21:35:57] RECOVERY - mw132 conntrack_table_size on mw132 is OK: OK: nf_conntrack is 0 % full [21:36:06] RECOVERY - mw132 ferm_active on mw132 is OK: OK ferm input default policy is set [21:36:11] RECOVERY - wikimoma.art - LetsEncrypt on sslhost is OK: OK - Certificate 'wikimoma.art' will expire on Sat 17 Sep 2022 19:04:53 GMT +0000. [21:36:29] RECOVERY - wiki.nevillepedia.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nevillepedia.eu' will expire on Wed 24 Aug 2022 04:41:20 GMT +0000. [21:36:41] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 5.426 second response time [21:36:44] RECOVERY - taswin.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000. [21:36:55] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 4.460 second response time [21:36:56] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15946 bytes in 3.014 second response time [21:37:02] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.581 second response time [21:37:05] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.72, 3.64, 3.65 [21:37:21] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Tue 23 Aug 2022 01:05:30 GMT +0000. [21:39:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/c002545d6e78 [21:39:58] [02miraheze/puppet] 07paladox 03c002545 - mw1* increase php childs to 32 [21:40:00] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [21:40:01] [02puppet] 07paladox opened pull request 03#2723: mw1* increase php childs to 32 - 13https://github.com/miraheze/puppet/pull/2723 [21:40:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c002545d6e78...6d7d90a57192 [21:40:20] [02miraheze/puppet] 07paladox 036d7d90a - Update mw102.yaml [21:40:22] [02puppet] 07paladox synchronize pull request 03#2723: mw1* increase php childs to 32 - 13https://github.com/miraheze/puppet/pull/2723 [21:40:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6d7d90a57192...59d3fed1a94c [21:40:30] [02miraheze/puppet] 07paladox 0359d3fed - Update mw111.yaml [21:40:32] [02puppet] 07paladox synchronize pull request 03#2723: mw1* increase php childs to 32 - 13https://github.com/miraheze/puppet/pull/2723 [21:40:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/59d3fed1a94c...500d411a28ef [21:40:42] [02miraheze/puppet] 07paladox 03500d411 - Update mw112.yaml [21:40:44] [02puppet] 07paladox synchronize pull request 03#2723: mw1* increase php childs to 32 - 13https://github.com/miraheze/puppet/pull/2723 [21:40:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/500d411a28ef...6b986b704c3e [21:40:54] [02miraheze/puppet] 07paladox 036b986b7 - Update mw121.yaml [21:40:56] [02puppet] 07paladox synchronize pull request 03#2723: mw1* increase php childs to 32 - 13https://github.com/miraheze/puppet/pull/2723 [21:41:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6b986b704c3e...83da88020e1f [21:41:21] [02miraheze/puppet] 07paladox 0383da880 - Update mw122.yaml [21:41:22] [02puppet] 07paladox synchronize pull request 03#2723: mw1* increase php childs to 32 - 13https://github.com/miraheze/puppet/pull/2723 [21:41:34] paladox: are you sure that will work [21:41:36] [02puppet] 07paladox closed pull request 03#2723: mw1* increase php childs to 32 - 13https://github.com/miraheze/puppet/pull/2723 [21:41:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://github.com/miraheze/puppet/compare/d06753bf9cc0...d23298a8239f [21:41:39] [02miraheze/puppet] 07paladox 03d23298a - mw1* increase php childs to 32 (#2723) [21:41:40] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [21:41:42] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [21:41:55] A high php child count last time just left it OOMing that much it was unstable [21:44:47] yeh, if it does cause major issues we can revert but i had this set for a bit as i hacked puppet but i resetted the config so lost it. [21:45:02] paladox: ok [21:46:58] [discord] . [21:50:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/191382482421 [21:50:31] [02miraheze/puppet] 07paladox 031913824 - Install mw131 and mw132 [21:50:32] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [21:50:34] [02puppet] 07paladox opened pull request 03#2724: Install mw131 and mw132 - 13https://github.com/miraheze/puppet/pull/2724 [21:50:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/191382482421...369115d90bb4 [21:50:47] [02miraheze/puppet] 07paladox 03369115d - Create mw132.yaml [21:50:49] [02puppet] 07paladox synchronize pull request 03#2724: Install mw131 and mw132 - 13https://github.com/miraheze/puppet/pull/2724 [21:51:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/369115d90bb4...e6339cb4fb1f [21:51:14] [02miraheze/puppet] 07paladox 03e6339cb - Update site.pp [21:51:15] [02puppet] 07paladox synchronize pull request 03#2724: Install mw131 and mw132 - 13https://github.com/miraheze/puppet/pull/2724 [21:51:22] [02puppet] 07paladox closed pull request 03#2724: Install mw131 and mw132 - 13https://github.com/miraheze/puppet/pull/2724 [21:51:24] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [21:51:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±1] 13https://github.com/miraheze/puppet/compare/d23298a8239f...ff61efcd82b6 [21:51:27] [02miraheze/puppet] 07paladox 03ff61efc - Install mw131 and mw132 (#2724) [21:51:28] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [21:52:12] [url] Page not found · GitHub · GitHub | github.com [21:52:13] [url] Comparing d23298a8239f...ff61efcd82b6 · miraheze/puppet · GitHub | github.com [21:52:13] [url] Page not found · GitHub · GitHub | github.com [21:54:46] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/1e2802dcd203 [21:54:46] [url] Page not found · GitHub · GitHub | github.com [21:54:47] [02miraheze/ssl] 07Universal-Omega 031e2802d - Add arcanewaterswiki.org cert [21:54:49] [02ssl] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [21:54:49] [url] Page not found · GitHub · GitHub | github.com [21:54:50] [02ssl] 07Universal-Omega opened pull request 03#582: Add arcanewaterswiki.org cert - 13https://github.com/miraheze/ssl/pull/582 [21:54:51] [url] Page not found · GitHub · GitHub | github.com [21:55:19] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1e2802dcd203...26dabf0ef9b8 [21:55:19] [url] Comparing 1e2802dcd203...26dabf0ef9b8 · miraheze/ssl · GitHub | github.com [21:55:20] [02miraheze/ssl] 07Universal-Omega 0326dabf0 - Update certs.yaml [21:55:22] [02ssl] 07Universal-Omega synchronize pull request 03#582: Add arcanewaterswiki.org cert - 13https://github.com/miraheze/ssl/pull/582 [21:55:22] ... [21:55:55] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:55:55] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:14] [02miraheze/ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:56:15] [02ssl] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ssl [21:56:16] ... [21:56:17] [02ssl] 07Universal-Omega closed pull request 03#582: Add arcanewaterswiki.org cert - 13https://github.com/miraheze/ssl/pull/582 [21:56:17] [url] Page not found · GitHub · GitHub | github.com [21:56:18] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/d8b57c3699e4...93bb4391e8ec [21:56:19] [url] Comparing d8b57c3699e4...93bb4391e8ec · miraheze/ssl · GitHub | github.com [21:56:20] [02miraheze/ssl] 07Universal-Omega 0393bb439 - Add arcanewaterswiki.org cert (#582) [21:56:30] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 4.47, 3.43, 2.53 [21:56:34] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:43] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:45] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:57:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.02, 10.60, 9.78 [21:57:05] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:57:05] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:57:22] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:57:22] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:57:28] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:57:41] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 3.146 second response time [21:57:46] RECOVERY - mw112 NTP time on mw112 is OK: NTP OK: Offset -0.09554076195 secs [22:00:11] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.308 second response time [22:00:20] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.020 second response time [22:00:23] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.054 second response time [22:00:34] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 0.427 second response time [22:00:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.81, 12.84, 10.75 [22:01:02] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 2.317 second response time [22:01:08] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:18] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:27] PROBLEM - mw132 php-fpm on mw132 is UNKNOWN: NRPE: Command 'check_php-fpm' not defined [22:01:32] PROBLEM - mw131 php-fpm on mw131 is UNKNOWN: NRPE: Command 'check_php-fpm' not defined [22:01:38] PROBLEM - mw131 nutcracker process on mw131 is UNKNOWN: NRPE: Command 'check_nutcracker process' not defined [22:01:40] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:44] PROBLEM - mw132 JobRunner Service on mw132 is UNKNOWN: NRPE: Command 'check_JobRunner Service' not defined [22:01:44] PROBLEM - mw131 JobRunner Service on mw131 is UNKNOWN: NRPE: Command 'check_JobRunner Service' not defined [22:01:49] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:50] PROBLEM - mw132 Check Gluster Clients on mw132 is UNKNOWN: NRPE: Command 'check_Check Gluster Clients' not defined [22:01:51] PROBLEM - mw131 Check Gluster Clients on mw131 is UNKNOWN: NRPE: Command 'check_Check Gluster Clients' not defined [22:02:00] PROBLEM - mw132 nutcracker process on mw132 is UNKNOWN: NRPE: Command 'check_nutcracker process' not defined [22:02:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 21.36, 14.36, 11.35 [22:02:06] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.59, 3.33, 3.32 [22:02:23] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:26] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw122 [22:02:26] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 7.470 second response time [22:02:30] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.332 second response time [22:02:50] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.023 second response time [22:02:55] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 0.020 second response time [22:03:00] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 19.92, 15.57, 11.63 [22:03:08] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.263 second response time [22:03:10] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 28487 bytes in 0.367 second response time [22:03:21] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 4.918 second response time [22:03:21] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.845 second response time [22:03:22] RECOVERY - mw132 php-fpm on mw132 is OK: PROCS OK: 33 processes with command name 'php-fpm7.4' [22:03:41] RECOVERY - mw132 JobRunner Service on mw132 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [22:03:44] RECOVERY - mw132 HTTPS on mw132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.012 second response time [22:03:49] PROBLEM - mw132 Check Gluster Clients on mw132 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [22:03:59] RECOVERY - mw132 nutcracker process on mw132 is OK: PROCS OK: 1 process with UID = 115 (nutcracker), command name 'nutcracker' [22:04:04] PROBLEM - mw132 MediaWiki Rendering on mw132 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 251 bytes in 0.009 second response time [22:04:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.68, 3.22, 3.30 [22:04:21] PROBLEM - mw131 MediaWiki Rendering on mw131 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 251 bytes in 0.018 second response time [22:05:10] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:26] RECOVERY - mw131 php-fpm on mw131 is OK: PROCS OK: 33 processes with command name 'php-fpm7.4' [22:05:27] RECOVERY - mw131 nutcracker process on mw131 is OK: PROCS OK: 1 process with UID = 115 (nutcracker), command name 'nutcracker' [22:05:42] RECOVERY - mw131 JobRunner Service on mw131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [22:05:46] RECOVERY - mw131 Check Gluster Clients on mw131 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [22:05:49] RECOVERY - mw131 HTTPS on mw131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 545 bytes in 0.014 second response time [22:06:08] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.08, 1.77, 1.20 [22:06:18] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 14 backends are healthy [22:06:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.60, 11.16, 10.95 [22:06:24] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:06:55] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:06:55] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/w/skins/Femiwiki/node_modules],Service[cgroup],Mount[/mnt/mediawiki-static] [22:07:06] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:07:15] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 6.993 second response time [22:07:16] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:07:48] RECOVERY - mw132 Check Gluster Clients on mw132 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [22:08:06] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.50, 1.35, 1.11 [22:08:19] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.306 second response time [22:08:34] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.68, 3.98, 3.55 [22:08:51] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28488 bytes in 1.568 second response time [22:09:10] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 3.681 second response time [22:09:36] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 21.99, 14.22, 10.36 [22:09:56] PROBLEM - mw131 Puppet on mw131 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown): Package[firejail],File[/etc/firejail/mediawiki.local],File[/etc/firejail/mediawiki-converters.profile],File[/srv/mediawiki/w/skins/Femiwiki/node_modules] [22:10:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 20.46, 13.93, 11.95 [22:11:22] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.315 second response time [22:12:25] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.47, 3.27, 3.37 [22:15:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.11, 11.76, 10.71 [22:16:10] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.59, 3.41, 3.17 [22:16:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.89, 11.35, 11.71 [22:16:33] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.53, 11.05, 11.79 [22:18:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.61, 3.57, 3.27 [22:18:24] !log [paladox@mwtask111] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw131.miraheze.org [22:19:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.39, 9.61, 10.05 [22:19:57] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.56, 11.56, 11.91 [22:20:07] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.32, 3.37, 3.22 [22:20:12] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 241 bytes in 0.024 second response time [22:20:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.41, 10.75, 11.31 [22:21:44] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 241 bytes in 0.015 second response time [22:22:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.19, 9.82, 10.91 [22:22:48] !log [paladox@mwtask111] finished deploy of {'config': True, 'world': True, 'force': True, 'servers': 'mw131,mw132'} to mw131,mw132 - SUCCESS in 256s [22:23:21] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.21, 10.04, 10.21 [22:25:47] [02miraheze/CreateWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/commit/cd01054a437f [22:25:49] [02miraheze/CreateWiki] 07dependabot[bot] 03cd01054 - Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 [22:25:50] [02CreateWiki] 07dependabot[bot] opened pull request 03#330: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/CreateWiki/pull/330 [22:25:51] [url] Page not found · GitHub · GitHub | github.com [22:25:52] [02CreateWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 - 13https://github.com/miraheze/CreateWiki [22:25:52] [url] Page not found · GitHub · GitHub | github.com [22:25:53] [02CreateWiki] 07dependabot[bot] labeled pull request 03#330: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/CreateWiki/pull/330 [22:25:54] [url] Page not found · GitHub · GitHub | github.com [22:25:55] [02CreateWiki] 07dependabot[bot] labeled pull request 03#330: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/CreateWiki/pull/330 [22:25:55] [url] Page not found · GitHub · GitHub | github.com [22:26:40] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/cd01054a437f...ff87fb993deb [22:26:41] [url] Comparing cd01054a437f...ff87fb993deb · miraheze/CreateWiki · GitHub | github.com [22:26:42] [02miraheze/CreateWiki] 07Universal-Omega 03ff87fb9 - Update .stylelintrc.json [22:26:43] [02CreateWiki] 07Universal-Omega synchronize pull request 03#330: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/CreateWiki/pull/330 [22:26:44] ... [22:27:36] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.73, 3.44, 2.80 [22:27:39] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.51, 11.64, 11.70 [22:27:44] [02CreateWiki] 07Universal-Omega closed pull request 03#330: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/CreateWiki/pull/330 [22:27:45] ... [22:27:46] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/CreateWiki/compare/3d7f9ec546cf...fd29ec311c8f [22:27:46] [url] Comparing 3d7f9ec546cf...fd29ec311c8f · miraheze/CreateWiki · GitHub | github.com [22:27:47] [02miraheze/CreateWiki] 07dependabot[bot] 03fd29ec3 - Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 (#330) [22:27:49] [02CreateWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 - 13https://github.com/miraheze/CreateWiki [22:27:49] [url] Page not found · GitHub · GitHub | github.com [22:27:50] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 [22:31:02] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.27, 11.40, 11.06 [22:31:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.24, 10.98, 11.41 [22:31:27] [02miraheze/IncidentReporting] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 [+0/-0/±2] 13https://github.com/miraheze/IncidentReporting/commit/33ab7cfdb797 [22:31:28] [url] Page not found · GitHub · GitHub | github.com [22:31:29] [02miraheze/IncidentReporting] 07dependabot[bot] 0333ab7cf - Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 [22:31:30] [02IncidentReporting] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 - 13https://github.com/miraheze/IncidentReporting [22:31:31] [url] Page not found · GitHub · GitHub | github.com [22:31:32] [02IncidentReporting] 07dependabot[bot] opened pull request 03#47: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/IncidentReporting/pull/47 [22:31:32] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.91, 3.38, 2.93 [22:31:32] [url] Page not found · GitHub · GitHub | github.com [22:31:33] [02IncidentReporting] 07dependabot[bot] labeled pull request 03#47: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/IncidentReporting/pull/47 [22:31:33] [url] Page not found · GitHub · GitHub | github.com [22:31:35] [02IncidentReporting] 07dependabot[bot] labeled pull request 03#47: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/IncidentReporting/pull/47 [22:31:35] ... [22:32:02] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/33ab7cfdb797...de0cd5021944 [22:32:03] [url] Comparing 33ab7cfdb797...de0cd5021944 · miraheze/IncidentReporting · GitHub | github.com [22:32:04] [02miraheze/IncidentReporting] 07Universal-Omega 03de0cd50 - Update .stylelintrc.json [22:32:05] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#47: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/IncidentReporting/pull/47 [22:32:06] ... [22:32:18] [02IncidentReporting] 07Universal-Omega closed pull request 03#47: Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 - 13https://github.com/miraheze/IncidentReporting/pull/47 [22:32:18] ... [22:32:19] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/IncidentReporting/compare/5fee4ab301d3...f4ca5f4c1856 [22:32:20] [url] Comparing 5fee4ab301d3...f4ca5f4c1856 · miraheze/IncidentReporting · GitHub | github.com [22:32:21] [02miraheze/IncidentReporting] 07dependabot[bot] 03f4ca5f4 - Bump eslint-config-wikimedia from 0.22.1 to 0.23.0 (#47) [22:32:22] [02miraheze/IncidentReporting] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 [22:32:24] [02IncidentReporting] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.23.0 - 13https://github.com/miraheze/IncidentReporting [22:32:24] [url] Page not found · GitHub · GitHub | github.com [22:32:50] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.50, 4.10, 3.57 [22:33:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.19, 10.21, 10.67 [22:34:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.46, 3.64, 3.48 [22:35:07] miraheze/CreateWiki - Universal-Omega the build has errored. [22:35:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.03, 12.20, 10.93 [22:35:25] PROBLEM - matomo101 Current Load on matomo101 is WARNING: WARNING - load average: 3.36, 3.47, 3.08 [22:35:31] miraheze/IncidentReporting - Universal-Omega the build passed. [22:36:43] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.61, 3.27, 3.36 [22:37:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.74, 11.05, 10.86 [22:37:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.04, 11.45, 10.83 [22:37:19] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 2.72, 3.24, 3.04 [22:37:20] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.48, 9.64, 10.17 [22:37:20] miraheze/CreateWiki - Universal-Omega the build passed. [22:37:47] miraheze/IncidentReporting - Universal-Omega the build passed. [22:38:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.81, 10.16, 10.56 [22:40:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.10, 12.11, 11.22 [22:41:00] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.48, 4.04, 3.03 [22:41:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.61, 12.20, 11.20 [22:41:15] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.66, 11.38, 10.77 [22:42:56] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 2.52, 3.32, 2.88 [22:43:02] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.02, 10.09, 10.42 [22:43:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.95, 11.51, 11.06 [22:44:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.38, 11.67, 11.34 [22:45:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.05, 9.75, 10.27 [22:46:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.18, 3.72, 3.38 [22:47:02] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.76, 9.33, 10.07 [22:47:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.22, 11.62, 11.18 [22:48:22] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.31, 3.34, 3.27 [22:48:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.24, 12.47, 11.70 [22:49:03] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.97, 11.69, 10.93 [22:50:43] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 3.84, 3.73, 3.12 [22:51:00] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.95, 10.92, 10.72 [22:51:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.28, 11.32, 10.82 [22:51:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.00, 11.17, 11.23 [22:52:39] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.05, 4.07, 3.30 [22:52:56] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.86, 9.37, 10.20 [22:55:02] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.31, 8.79, 9.93 [22:55:30] PROBLEM - cp21 Varnish Backends on cp21 is CRITICAL: 1 backends are down. mw112 [22:55:41] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:56] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:08] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 1 backends are down. mw112 [22:57:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.50, 11.58, 11.34 [22:57:26] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 14 backends are healthy [22:57:26] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:36] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 0.308 second response time [22:57:40] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:53] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:58] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 7.989 second response time [22:58:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.85, 10.19, 12.00 [22:58:39] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:58:56] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.520 second response time [22:59:02] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:59:04] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:59:08] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 14 backends are healthy [22:59:15] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:59:39] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.028 second response time [22:59:52] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.342 second response time [23:00:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.97, 11.52, 11.73 [23:01:01] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.485 second response time [23:01:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.10, 11.82, 11.55 [23:02:15] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 9.616 second response time [23:02:43] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 28479 bytes in 2.839 second response time [23:04:49] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.43, 11.53, 11.68 [23:06:48] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.43, 10.40, 11.20 [23:07:19] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.923 second response time [23:07:23] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 1.220 second response time [23:07:35] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.017 second response time [23:09:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.56, 3.44, 3.22 [23:10:04] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.69, 3.19, 3.84 [23:10:46] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.77, 11.95, 11.58 [23:11:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.26, 3.72, 3.35 [23:12:07] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 15.17, 11.86, 11.41 [23:13:39] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.40, 3.65, 3.37 [23:14:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.79, 11.47, 11.32 [23:14:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.04, 11.99, 11.76 [23:15:31] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:35] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.42, 3.32, 3.28 [23:15:37] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:52] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.73, 3.93, 3.89 [23:16:07] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.88, 11.68, 11.40 [23:16:42] !log [paladox@mwtask111] finished deploy of {'landing': True, 'force': True, 'servers': 'mw131,mw132'} to mw131,mw132 - SUCCESS in 8s [23:16:44] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.00, 12.69, 12.06 [23:17:26] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.313 second response time [23:17:38] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.89, 1.99, 1.50 [23:17:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.29, 10.01, 9.51 [23:18:31] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [23:18:33] PROBLEM - mw132 Check Gluster Clients on mw132 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [23:18:45] PROBLEM - mw131 Check Gluster Clients on mw131 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [23:18:49] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:19:41] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 15938 bytes in 1.448 second response time [23:19:44] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.88, 3.70, 3.83 [23:19:49] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.15, 12.14, 10.39 [23:19:50] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:20:07] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.05, 11.39, 11.46 [23:20:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.42, 11.55, 11.79 [23:20:43] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.018 second response time [23:20:45] !log [paladox@mwtask111] finished deploy of {'errorpages': True, 'force': True, 'servers': 'mw131,mw132'} to mw131,mw132 - SUCCESS in 2s [23:20:50] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 24988 bytes in 0.440 second response time [23:20:55] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 24988 bytes in 0.401 second response time [23:23:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.08, 11.81, 10.65 [23:23:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/e70e8ccbec9d [23:23:45] [02miraheze/puppet] 07paladox 03e70e8cc - Pool in mw131 and mw132 [23:23:47] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [23:23:47] [url] Page not found · GitHub · GitHub | github.com [23:23:48] [02puppet] 07paladox opened pull request 03#2725: Pool in mw131 and mw132 - 13https://github.com/miraheze/puppet/pull/2725 [23:23:49] [url] Page not found · GitHub · GitHub | github.com [23:24:05] [02puppet] 07paladox closed pull request 03#2725: Pool in mw131 and mw132 - 13https://github.com/miraheze/puppet/pull/2725 [23:24:05] [url] Page not found · GitHub · GitHub | github.com [23:24:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ff61efcd82b6...a7a87444ea64 [23:24:07] [url] Comparing ff61efcd82b6...a7a87444ea64 · miraheze/puppet · GitHub | github.com [23:24:08] [02miraheze/puppet] 07paladox 03a7a8744 - Pool in mw131 and mw132 (#2725) [23:24:09] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [23:24:11] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/puppet [23:24:11] [url] Page not found · GitHub · GitHub | github.com [23:24:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.68, 3.91, 3.57 [23:25:36] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 17.25, 12.96, 11.15 [23:26:17] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.40, 3.30, 3.38 [23:26:26] RECOVERY - mw132 Check Gluster Clients on mw132 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [23:26:45] RECOVERY - mw131 Check Gluster Clients on mw131 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [23:27:14] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.15, 10.66, 10.56 [23:27:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.70, 1.90, 1.83 [23:28:07] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.98, 12.42, 11.72 [23:30:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.27, 9.15, 10.03 [23:31:02] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.62, 10.71, 11.13 [23:31:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.51, 10.48, 10.74 [23:31:20] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 7.11, 4.42, 3.95 [23:31:57] PROBLEM - cp20 Stunnel HTTP for mw131 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:32:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.48, 8.06, 9.92 [23:32:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.47, 1.55, 1.70 [23:32:58] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.40, 8.85, 10.12 [23:33:54] RECOVERY - cp20 Stunnel HTTP for mw131 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 15952 bytes in 0.021 second response time [23:34:14] !log mon111: upgrade grafana to 9.0.5 [23:36:43] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.13, 1.63, 1.74 [23:37:28] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.48, 3.69, 3.55 [23:38:40] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.01, 1.48, 1.67 [23:38:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.09, 8.77, 10.16 [23:39:00] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 2.59, 3.79, 3.98 [23:39:26] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 3.84, 3.61 [23:40:55] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 5.58, 4.34, 4.13 [23:42:50] PROBLEM - gluster122 Current Load on gluster122 is WARNING: WARNING - load average: 1.72, 3.30, 3.78 [23:46:52] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/3db6ca4827c9 [23:46:53] [02miraheze/dns] 07paladox 033db6ca4 - Add mem131 to dns [23:46:55] [02dns] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/dns [23:46:55] [url] Page not found · GitHub · GitHub | github.com [23:46:56] [02dns] 07paladox opened pull request 03#314: Add mem131 to dns - 13https://github.com/miraheze/dns/pull/314 [23:46:57] ... [23:47:01] [02dns] 07paladox closed pull request 03#314: Add mem131 to dns - 13https://github.com/miraheze/dns/pull/314 [23:47:02] ... [23:47:03] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/03525afeb797...b84fa00112d5 [23:47:03] [url] Comparing 03525afeb797...b84fa00112d5 · miraheze/dns · GitHub | github.com [23:47:04] [02miraheze/dns] 07paladox 03b84fa00 - Add mem131 to dns (#314) [23:47:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.53, 3.60, 3.64 [23:48:34] RECOVERY - gluster122 Current Load on gluster122 is OK: OK - load average: 1.88, 2.48, 3.27 [23:48:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.38, 9.39, 11.92 [23:48:50] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-3 [23:48:52] [02dns] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/dns [23:48:52] [url] Page not found · GitHub · GitHub | github.com [23:57:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.42, 3.43, 3.50 [23:58:09] PROBLEM - gluster122 Current Load on gluster122 is CRITICAL: CRITICAL - load average: 6.48, 4.29, 3.61 [23:58:59] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.29, 3.37, 3.47