[00:00:59] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.27, 9.56, 9.24 [00:01:04] [02mw-config] 07Universal-Omega closed pull request 03#4394: Fix inkscape - 13https://github.com/miraheze/mw-config/pull/4394 [00:01:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8dac6280fbcc...8e5e4c3273c0 [00:01:07] [02miraheze/mw-config] 07Universal-Omega 038e5e4c3 - Fix inkscape (#4394) [00:01:08] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [00:01:10] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 [00:01:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.54, 9.06, 9.75 [00:02:11] miraheze/mw-config - Universal-Omega the build passed. [00:02:16] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.79, 9.51, 9.89 [00:04:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.89, 9.51, 9.26 [00:04:52] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.28, 9.81, 9.89 [00:06:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.57, 3.70, 3.56 [00:07:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.57, 10.54, 9.82 [00:07:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.31, 3.78, 3.43 [00:07:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.24, 9.16, 8.79 [00:07:43] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 374 failures. Last run 7 minutes ago with 374 failures. Failed resources (up to 3 shown): File[wiki.jumpbound.com],File[wiki.jumpbound.com_private],File[wiki.astralprojections.org],File[wiki.astralprojections.org_private] [00:08:50] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.99, 10.18, 10.04 [00:09:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.48, 3.25, 3.42 [00:10:24] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.40, 3.17, 3.26 [00:12:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.96, 9.83, 9.90 [00:12:49] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.39, 2.95, 3.27 [00:13:44] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.22, 8.77, 8.69 [00:17:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.51, 9.50, 9.29 [00:18:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.23, 10.23, 10.00 [00:18:46] !log [@test101] starting deploy of {'config': True} to skip [00:18:47] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [00:19:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:21] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:20:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:20:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.00, 9.73, 9.48 [00:20:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [00:20:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.69, 3.52 [00:21:06] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.22, 3.85, 3.63 [00:21:57] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.64, 8.87, 8.51 [00:24:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.84, 4.80, 3.96 [00:24:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.57, 3.95, 3.70 [00:24:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.10, 4.64, 3.77 [00:26:31] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [00:26:32] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:26:33] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.91, 9.87, 9.32 [00:26:54] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:27:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.80, 3.50, 3.55 [00:27:35] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [00:28:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.40, 8.41, 8.46 [00:29:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.20, 9.71, 9.36 [00:30:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.12, 3.79, 3.79 [00:30:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.69, 3.60 [00:32:32] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:32:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.05, 2.82, 3.25 [00:33:07] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.72, 2.78, 3.26 [00:39:05] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.21, 2.95, 3.34 [00:41:04] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.79, 9.44, 8.78 [00:44:08] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.39, 9.18, 8.80 [00:44:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.03, 9.43, 9.07 [00:46:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:46:58] [02mediawiki] 07dependabot[bot] opened pull request 03#4942: Bump extensions/FlaggedRevs from `f3a8ffc` to `7aaa50a` - 13https://github.com/miraheze/mediawiki/pull/4942 [00:47:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/FlaggedRevs-7aaa50a [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0ec21e415785 [00:47:01] [02miraheze/mediawiki] 07dependabot[bot] 030ec21e4 - Bump extensions/FlaggedRevs from `f3a8ffc` to `7aaa50a` [00:47:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/FlaggedRevs-7aaa50a - 13https://github.com/miraheze/mediawiki [00:47:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AbuseFilter-7d92a02 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/02ae3324d2c3 [00:47:06] [02miraheze/mediawiki] 07dependabot[bot] 0302ae332 - Bump extensions/AbuseFilter from `601ffab` to `7d92a02` [00:47:07] [02mediawiki] 07dependabot[bot] opened pull request 03#4943: Bump extensions/AbuseFilter from `601ffab` to `7d92a02` - 13https://github.com/miraheze/mediawiki/pull/4943 [00:47:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AbuseFilter-7d92a02 - 13https://github.com/miraheze/mediawiki [00:47:10] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Kartographer-d17fce4 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/35009d017632 [00:47:12] [02miraheze/mediawiki] 07dependabot[bot] 0335009d0 - Bump extensions/Kartographer from `3ba56a9` to `d17fce4` [00:47:13] [02mediawiki] 07dependabot[bot] opened pull request 03#4944: Bump extensions/Kartographer from `3ba56a9` to `d17fce4` - 13https://github.com/miraheze/mediawiki/pull/4944 [00:47:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Kartographer-d17fce4 - 13https://github.com/miraheze/mediawiki [00:47:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MassMessage-2850498 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2e68550c2c83 [00:47:18] [02miraheze/mediawiki] 07dependabot[bot] 032e68550 - Bump extensions/MassMessage from `7e532a9` to `2850498` [00:47:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MassMessage-2850498 - 13https://github.com/miraheze/mediawiki [00:47:21] [02mediawiki] 07dependabot[bot] opened pull request 03#4945: Bump extensions/MassMessage from `7e532a9` to `2850498` - 13https://github.com/miraheze/mediawiki/pull/4945 [00:47:22] [02mediawiki] 07dependabot[bot] opened pull request 03#4946: Bump extensions/LinkTitles from `d6dd9eb` to `4e868e3` - 13https://github.com/miraheze/mediawiki/pull/4946 [00:47:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/LinkTitles-4e868e3 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ba3db7454ec0 [00:47:25] [02miraheze/mediawiki] 07dependabot[bot] 03ba3db74 - Bump extensions/LinkTitles from `d6dd9eb` to `4e868e3` [00:47:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/LinkTitles-4e868e3 - 13https://github.com/miraheze/mediawiki [00:47:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalPreferences-ffb96a7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/14caac9dde80 [00:47:30] [02miraheze/mediawiki] 07dependabot[bot] 0314caac9 - Bump extensions/GlobalPreferences from `8e54bed` to `ffb96a7` [00:47:31] [02mediawiki] 07dependabot[bot] opened pull request 03#4947: Bump extensions/GlobalPreferences from `8e54bed` to `ffb96a7` - 13https://github.com/miraheze/mediawiki/pull/4947 [00:47:32] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.82, 9.86, 9.13 [00:47:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalPreferences-ffb96a7 - 13https://github.com/miraheze/mediawiki [00:47:34] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ApprovedRevs-53e11db [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9ac052bcd123 [00:47:36] [02miraheze/mediawiki] 07dependabot[bot] 039ac052b - Bump extensions/ApprovedRevs from `083cc00` to `53e11db` [00:47:37] [02mediawiki] 07dependabot[bot] opened pull request 03#4948: Bump extensions/ApprovedRevs from `083cc00` to `53e11db` - 13https://github.com/miraheze/mediawiki/pull/4948 [00:47:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ApprovedRevs-53e11db - 13https://github.com/miraheze/mediawiki [00:47:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalWatchlist-663a1ec [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0a7529769613 [00:47:42] [02miraheze/mediawiki] 07dependabot[bot] 030a75297 - Bump extensions/GlobalWatchlist from `d302af9` to `663a1ec` [00:47:43] [02mediawiki] 07dependabot[bot] opened pull request 03#4949: Bump extensions/GlobalWatchlist from `d302af9` to `663a1ec` - 13https://github.com/miraheze/mediawiki/pull/4949 [00:47:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalWatchlist-663a1ec - 13https://github.com/miraheze/mediawiki [00:47:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#4942: Bump extensions/FlaggedRevs from `f3a8ffc` to `7aaa50a` - 13https://github.com/miraheze/mediawiki/pull/4942 [00:47:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MintyDocs-cf972cd [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/511eaf8ab7ee [00:47:49] [02miraheze/mediawiki] 07dependabot[bot] 03511eaf8 - Bump extensions/MintyDocs from `f34dd30` to `cf972cd` [00:47:51] [02mediawiki] 07dependabot[bot] opened pull request 03#4950: Bump extensions/MintyDocs from `f34dd30` to `cf972cd` - 13https://github.com/miraheze/mediawiki/pull/4950 [00:47:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MintyDocs-cf972cd - 13https://github.com/miraheze/mediawiki [00:47:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#4944: Bump extensions/Kartographer from `3ba56a9` to `d17fce4` - 13https://github.com/miraheze/mediawiki/pull/4944 [00:47:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AntiSpoof-37c00a4 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/118aa7f9e3e5 [00:47:57] [02miraheze/mediawiki] 07dependabot[bot] 03118aa7f - Bump extensions/AntiSpoof from `ed0a896` to `37c00a4` [00:47:58] [02mediawiki] 07dependabot[bot] opened pull request 03#4951: Bump extensions/AntiSpoof from `ed0a896` to `37c00a4` - 13https://github.com/miraheze/mediawiki/pull/4951 [00:48:00] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AntiSpoof-37c00a4 - 13https://github.com/miraheze/mediawiki [00:48:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#4943: Bump extensions/AbuseFilter from `601ffab` to `7d92a02` - 13https://github.com/miraheze/mediawiki/pull/4943 [00:48:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#4945: Bump extensions/MassMessage from `7e532a9` to `2850498` - 13https://github.com/miraheze/mediawiki/pull/4945 [00:48:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/RegexFunctions-b01c348 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/413f1c1a4e08 [00:48:06] [02miraheze/mediawiki] 07dependabot[bot] 03413f1c1 - Bump extensions/RegexFunctions from `0075237` to `b01c348` [00:48:07] [02mediawiki] 07dependabot[bot] opened pull request 03#4952: Bump extensions/RegexFunctions from `0075237` to `b01c348` - 13https://github.com/miraheze/mediawiki/pull/4952 [00:48:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/RegexFunctions-b01c348 - 13https://github.com/miraheze/mediawiki [00:48:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#4947: Bump extensions/GlobalPreferences from `8e54bed` to `ffb96a7` - 13https://github.com/miraheze/mediawiki/pull/4947 [00:48:12] [02mediawiki] 07github-actions[bot] labeled pull request 03#4947: Bump extensions/GlobalPreferences from `8e54bed` to `ffb96a7` - 13https://github.com/miraheze/mediawiki/pull/4947 [00:48:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#4948: Bump extensions/ApprovedRevs from `083cc00` to `53e11db` - 13https://github.com/miraheze/mediawiki/pull/4948 [00:48:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#4946: Bump extensions/LinkTitles from `d6dd9eb` to `4e868e3` - 13https://github.com/miraheze/mediawiki/pull/4946 [00:48:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CommentStreams-fee8267 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/3f973cdad2f7 [00:48:18] [02miraheze/mediawiki] 07dependabot[bot] 033f973cd - Bump extensions/CommentStreams from `702ce1c` to `fee8267` [00:48:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#4948: Bump extensions/ApprovedRevs from `083cc00` to `53e11db` - 13https://github.com/miraheze/mediawiki/pull/4948 [00:48:21] [02mediawiki] 07dependabot[bot] opened pull request 03#4953: Bump extensions/CommentStreams from `702ce1c` to `fee8267` - 13https://github.com/miraheze/mediawiki/pull/4953 [00:48:21] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.97, 10.09, 9.62 [00:48:22] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CommentStreams-fee8267 - 13https://github.com/miraheze/mediawiki [00:48:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GuidedTour-8a53adb [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8634669b4b00 [00:48:25] [02miraheze/mediawiki] 07dependabot[bot] 038634669 - Bump extensions/GuidedTour from `03da72b` to `8a53adb` [00:48:27] [02mediawiki] 07dependabot[bot] opened pull request 03#4954: Bump extensions/GuidedTour from `03da72b` to `8a53adb` - 13https://github.com/miraheze/mediawiki/pull/4954 [00:48:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GuidedTour-8a53adb - 13https://github.com/miraheze/mediawiki [00:48:30] [02mediawiki] 07dependabot[bot] opened pull request 03#4955: Bump extensions/Flow from `8643327` to `bd32d47` - 13https://github.com/miraheze/mediawiki/pull/4955 [00:48:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Flow-bd32d47 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/668fd4aeede7 [00:48:33] [02miraheze/mediawiki] 07dependabot[bot] 03668fd4a - Bump extensions/Flow from `8643327` to `bd32d47` [00:48:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Flow-bd32d47 - 13https://github.com/miraheze/mediawiki [00:48:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Disambiguator-7275829 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/034581b9992c [00:48:37] [02miraheze/mediawiki] 07dependabot[bot] 03034581b - Bump extensions/Disambiguator from `8cb42d8` to `7275829` [00:48:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#4949: Bump extensions/GlobalWatchlist from `d302af9` to `663a1ec` - 13https://github.com/miraheze/mediawiki/pull/4949 [00:48:40] [02mediawiki] 07dependabot[bot] opened pull request 03#4956: Bump extensions/Disambiguator from `8cb42d8` to `7275829` - 13https://github.com/miraheze/mediawiki/pull/4956 [00:48:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Disambiguator-7275829 - 13https://github.com/miraheze/mediawiki [00:48:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#4950: Bump extensions/MintyDocs from `f34dd30` to `cf972cd` - 13https://github.com/miraheze/mediawiki/pull/4950 [00:48:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PollNY-f22c8b2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/57a03d228e40 [00:48:46] [02miraheze/mediawiki] 07dependabot[bot] 0357a03d2 - Bump extensions/PollNY from `1d8efb8` to `f22c8b2` [00:48:48] [02mediawiki] 07dependabot[bot] opened pull request 03#4957: Bump extensions/PollNY from `1d8efb8` to `f22c8b2` - 13https://github.com/miraheze/mediawiki/pull/4957 [00:48:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PollNY-f22c8b2 - 13https://github.com/miraheze/mediawiki [00:48:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Cite-976ac42 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b0428d84c931 [00:48:52] [02miraheze/mediawiki] 07dependabot[bot] 03b0428d8 - Bump extensions/Cite from `797d179` to `976ac42` [00:48:54] [02mediawiki] 07dependabot[bot] opened pull request 03#4958: Bump extensions/Cite from `797d179` to `976ac42` - 13https://github.com/miraheze/mediawiki/pull/4958 [00:48:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#4951: Bump extensions/AntiSpoof from `ed0a896` to `37c00a4` - 13https://github.com/miraheze/mediawiki/pull/4951 [00:48:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Cite-976ac42 - 13https://github.com/miraheze/mediawiki [00:48:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#4953: Bump extensions/CommentStreams from `702ce1c` to `fee8267` - 13https://github.com/miraheze/mediawiki/pull/4953 [00:49:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/RemovePII-1b47a35 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4fde44ac6e2d [00:49:01] [02miraheze/mediawiki] 07dependabot[bot] 034fde44a - Bump extensions/RemovePII from `8c799ed` to `1b47a35` [00:49:03] [02mediawiki] 07dependabot[bot] opened pull request 03#4959: Bump extensions/RemovePII from `8c799ed` to `1b47a35` - 13https://github.com/miraheze/mediawiki/pull/4959 [00:49:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#4954: Bump extensions/GuidedTour from `03da72b` to `8a53adb` - 13https://github.com/miraheze/mediawiki/pull/4954 [00:49:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/RemovePII-1b47a35 - 13https://github.com/miraheze/mediawiki [00:49:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#4952: Bump extensions/RegexFunctions from `0075237` to `b01c348` - 13https://github.com/miraheze/mediawiki/pull/4952 [00:49:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Thanks-0701bde [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/76c7edbb4806 [00:49:10] [02miraheze/mediawiki] 07dependabot[bot] 0376c7edb - Bump extensions/Thanks from `9421bd5` to `0701bde` [00:49:12] [02mediawiki] 07dependabot[bot] opened pull request 03#4960: Bump extensions/Thanks from `9421bd5` to `0701bde` - 13https://github.com/miraheze/mediawiki/pull/4960 [00:49:13] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Thanks-0701bde - 13https://github.com/miraheze/mediawiki [00:49:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/QuizGame-797bcc0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/fdc1afe307bf [00:49:16] [02miraheze/mediawiki] 07dependabot[bot] 03fdc1afe - Bump extensions/QuizGame from `120604a` to `797bcc0` [00:49:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/QuizGame-797bcc0 - 13https://github.com/miraheze/mediawiki [00:49:19] [02mediawiki] 07dependabot[bot] opened pull request 03#4961: Bump extensions/QuizGame from `120604a` to `797bcc0` - 13https://github.com/miraheze/mediawiki/pull/4961 [00:49:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TitleBlacklist-d8c4487 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/64e7cb12273a [00:49:22] [02miraheze/mediawiki] 07dependabot[bot] 0364e7cb1 - Bump extensions/TitleBlacklist from `326e3cf` to `d8c4487` [00:49:23] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to ovlon [00:49:24] [02mediawiki] 07dependabot[bot] opened pull request 03#4962: Bump extensions/TitleBlacklist from `326e3cf` to `d8c4487` - 13https://github.com/miraheze/mediawiki/pull/4962 [00:49:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TitleBlacklist-d8c4487 - 13https://github.com/miraheze/mediawiki [00:49:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ManageWiki-a5a55c2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/00dc7cc00187 [00:49:28] [02miraheze/mediawiki] 07dependabot[bot] 0300dc7cc - Bump extensions/ManageWiki from `dc6542e` to `a5a55c2` [00:49:30] [02mediawiki] 07dependabot[bot] opened pull request 03#4963: Bump extensions/ManageWiki from `dc6542e` to `a5a55c2` - 13https://github.com/miraheze/mediawiki/pull/4963 [00:49:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ManageWiki-a5a55c2 - 13https://github.com/miraheze/mediawiki [00:49:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#4955: Bump extensions/Flow from `8643327` to `bd32d47` - 13https://github.com/miraheze/mediawiki/pull/4955 [00:49:34] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CiteThisPage-0b662a9 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/782fed2df782 [00:49:36] [02miraheze/mediawiki] 07dependabot[bot] 03782fed2 - Bump extensions/CiteThisPage from `67239de` to `0b662a9` [00:49:37] [02mediawiki] 07dependabot[bot] opened pull request 03#4964: Bump extensions/CiteThisPage from `67239de` to `0b662a9` - 13https://github.com/miraheze/mediawiki/pull/4964 [00:49:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CiteThisPage-0b662a9 - 13https://github.com/miraheze/mediawiki [00:49:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#4956: Bump extensions/Disambiguator from `8cb42d8` to `7275829` - 13https://github.com/miraheze/mediawiki/pull/4956 [00:49:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/VisualEditor-b0b884f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/613c48251a2c [00:49:43] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to scsvg [00:49:43] [02miraheze/mediawiki] 07dependabot[bot] 03613c482 - Bump extensions/VisualEditor from `3aeea63` to `b0b884f` [00:49:45] [02mediawiki] 07dependabot[bot] opened pull request 03#4965: Bump extensions/VisualEditor from `3aeea63` to `b0b884f` - 13https://github.com/miraheze/mediawiki/pull/4965 [00:49:46] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/VisualEditor-b0b884f - 13https://github.com/miraheze/mediawiki [00:49:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/BetaFeatures-6dbf978 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9059c59f0332 [00:49:49] [02miraheze/mediawiki] 07dependabot[bot] 039059c59 - Bump extensions/BetaFeatures from `2820d7f` to `6dbf978` [00:49:51] [02mediawiki] 07dependabot[bot] opened pull request 03#4966: Bump extensions/BetaFeatures from `2820d7f` to `6dbf978` - 13https://github.com/miraheze/mediawiki/pull/4966 [00:49:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/BetaFeatures-6dbf978 - 13https://github.com/miraheze/mediawiki [00:49:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#4957: Bump extensions/PollNY from `1d8efb8` to `f22c8b2` - 13https://github.com/miraheze/mediawiki/pull/4957 [00:49:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#4960: Bump extensions/Thanks from `9421bd5` to `0701bde` - 13https://github.com/miraheze/mediawiki/pull/4960 [00:49:57] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/apex-e0eadb6 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f795cf3b8ef0 [00:49:58] [02miraheze/mediawiki] 07dependabot[bot] 03f795cf3 - Bump skins/apex from `8bf800a` to `e0eadb6` [00:50:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#4958: Bump extensions/Cite from `797d179` to `976ac42` - 13https://github.com/miraheze/mediawiki/pull/4958 [00:50:01] [02mediawiki] 07dependabot[bot] opened pull request 03#4967: Bump skins/apex from `8bf800a` to `e0eadb6` - 13https://github.com/miraheze/mediawiki/pull/4967 [00:50:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#4959: Bump extensions/RemovePII from `8c799ed` to `1b47a35` - 13https://github.com/miraheze/mediawiki/pull/4959 [00:50:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#4961: Bump extensions/QuizGame from `120604a` to `797bcc0` - 13https://github.com/miraheze/mediawiki/pull/4961 [00:50:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/apex-e0eadb6 - 13https://github.com/miraheze/mediawiki [00:50:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#4962: Bump extensions/TitleBlacklist from `326e3cf` to `d8c4487` - 13https://github.com/miraheze/mediawiki/pull/4962 [00:50:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.54, 3.51, 3.49 [00:50:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ProofreadPage-8204d15 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/40de9ec722a9 [00:50:10] [02miraheze/mediawiki] 07dependabot[bot] 0340de9ec - Bump extensions/ProofreadPage from `a4aedd2` to `8204d15` [00:50:12] [02mediawiki] 07dependabot[bot] opened pull request 03#4968: Bump extensions/ProofreadPage from `a4aedd2` to `8204d15` - 13https://github.com/miraheze/mediawiki/pull/4968 [00:50:13] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ProofreadPage-8204d15 - 13https://github.com/miraheze/mediawiki [00:50:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TranslationNotifications-e0cce2d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/40f71132e5bd [00:50:16] [02miraheze/mediawiki] 07dependabot[bot] 0340f7113 - Bump extensions/TranslationNotifications from `d13856f` to `e0cce2d` [00:50:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TranslationNotifications-e0cce2d - 13https://github.com/miraheze/mediawiki [00:50:19] [02mediawiki] 07dependabot[bot] opened pull request 03#4969: Bump extensions/TranslationNotifications from `d13856f` to `e0cce2d` - 13https://github.com/miraheze/mediawiki/pull/4969 [00:50:21] [02mediawiki] 07github-actions[bot] labeled pull request 03#4963: Bump extensions/ManageWiki from `dc6542e` to `a5a55c2` - 13https://github.com/miraheze/mediawiki/pull/4963 [00:50:22] [02mediawiki] 07dependabot[bot] opened pull request 03#4970: Bump extensions/Score from `932fe9e` to `4e642a6` - 13https://github.com/miraheze/mediawiki/pull/4970 [00:50:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Score-4e642a6 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1b197644387c [00:50:25] [02miraheze/mediawiki] 07dependabot[bot] 031b19764 - Bump extensions/Score from `932fe9e` to `4e642a6` [00:50:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Score-4e642a6 - 13https://github.com/miraheze/mediawiki [00:50:27] PROBLEM - cp31 Stunnel Http for mw122 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:50:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CheckUser-610ba68 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8c83b79504cd [00:50:30] [02miraheze/mediawiki] 07dependabot[bot] 038c83b79 - Bump extensions/CheckUser from `72187a6` to `610ba68` [00:50:31] [02mediawiki] 07dependabot[bot] opened pull request 03#4971: Bump extensions/CheckUser from `72187a6` to `610ba68` - 13https://github.com/miraheze/mediawiki/pull/4971 [00:50:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CheckUser-610ba68 - 13https://github.com/miraheze/mediawiki [00:50:34] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to scsvg - FAIL: [256, 0, 0, 0, 0, 0, 0, 0] in 51s [00:50:34] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-b707511 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9ed99301c3ae [00:50:36] [02miraheze/mediawiki] 07dependabot[bot] 039ed9930 - Bump extensions/UniversalLanguageSelector from `d2b84d7` to `b707511` [00:50:37] [02mediawiki] 07dependabot[bot] opened pull request 03#4972: Bump extensions/UniversalLanguageSelector from `d2b84d7` to `b707511` - 13https://github.com/miraheze/mediawiki/pull/4972 [00:50:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#4964: Bump extensions/CiteThisPage from `67239de` to `0b662a9` - 13https://github.com/miraheze/mediawiki/pull/4964 [00:50:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-b707511 - 13https://github.com/miraheze/mediawiki [00:50:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:50:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#4965: Bump extensions/VisualEditor from `3aeea63` to `b0b884f` - 13https://github.com/miraheze/mediawiki/pull/4965 [00:50:43] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Renameuser-74f89d1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9892b598150c [00:50:45] [02miraheze/mediawiki] 07dependabot[bot] 039892b59 - Bump extensions/Renameuser from `d112548` to `74f89d1` [00:50:46] [02mediawiki] 07dependabot[bot] opened pull request 03#4973: Bump extensions/Renameuser from `d112548` to `74f89d1` - 13https://github.com/miraheze/mediawiki/pull/4973 [00:50:48] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Renameuser-74f89d1 - 13https://github.com/miraheze/mediawiki [00:50:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#4966: Bump extensions/BetaFeatures from `2820d7f` to `6dbf978` - 13https://github.com/miraheze/mediawiki/pull/4966 [00:50:51] [02mediawiki] 07dependabot[bot] opened pull request 03#4974: Bump extensions/Babel from `5b01abf` to `559a00a` - 13https://github.com/miraheze/mediawiki/pull/4974 [00:50:52] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Babel-559a00a [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c392437b17f5 [00:50:54] [02miraheze/mediawiki] 07dependabot[bot] 03c392437 - Bump extensions/Babel from `5b01abf` to `559a00a` [00:50:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Babel-559a00a - 13https://github.com/miraheze/mediawiki [00:50:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#4967: Bump skins/apex from `8bf800a` to `e0eadb6` - 13https://github.com/miraheze/mediawiki/pull/4967 [00:50:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/StandardDialogs-73b2d03 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/cf8eb60fe636 [00:51:00] [02miraheze/mediawiki] 07dependabot[bot] 03cf8eb60 - Bump extensions/StandardDialogs from `f601fed` to `73b2d03` [00:51:01] [02mediawiki] 07dependabot[bot] opened pull request 03#4975: Bump extensions/StandardDialogs from `f601fed` to `73b2d03` - 13https://github.com/miraheze/mediawiki/pull/4975 [00:51:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/StandardDialogs-73b2d03 - 13https://github.com/miraheze/mediawiki [00:51:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.56, 8.51, 8.72 [00:51:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#4969: Bump extensions/TranslationNotifications from `d13856f` to `e0cce2d` - 13https://github.com/miraheze/mediawiki/pull/4969 [00:51:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalUsage-70afccc [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4c3f6a6ced40 [00:51:06] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:51:07] [02miraheze/mediawiki] 07dependabot[bot] 034c3f6a6 - Bump extensions/GlobalUsage from `87146f0` to `70afccc` [00:51:09] [02mediawiki] 07dependabot[bot] opened pull request 03#4976: Bump extensions/GlobalUsage from `87146f0` to `70afccc` - 13https://github.com/miraheze/mediawiki/pull/4976 [00:51:10] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalUsage-70afccc - 13https://github.com/miraheze/mediawiki [00:51:12] [02mediawiki] 07github-actions[bot] labeled pull request 03#4968: Bump extensions/ProofreadPage from `a4aedd2` to `8204d15` - 13https://github.com/miraheze/mediawiki/pull/4968 [00:51:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DiscussionTools-397361d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a6c85a689e63 [00:51:15] [02miraheze/mediawiki] 07dependabot[bot] 03a6c85a6 - Bump extensions/DiscussionTools from `2fcc0eb` to `397361d` [00:51:16] [02mediawiki] 07dependabot[bot] opened pull request 03#4977: Bump extensions/DiscussionTools from `2fcc0eb` to `397361d` - 13https://github.com/miraheze/mediawiki/pull/4977 [00:51:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DiscussionTools-397361d - 13https://github.com/miraheze/mediawiki [00:51:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#4971: Bump extensions/CheckUser from `72187a6` to `610ba68` - 13https://github.com/miraheze/mediawiki/pull/4971 [00:51:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalBlocking-b7441ba [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/04b9991a2774 [00:51:22] [02miraheze/mediawiki] 07dependabot[bot] 0304b9991 - Bump extensions/GlobalBlocking from `2981bfc` to `b7441ba` [00:51:24] [02mediawiki] 07github-actions[bot] labeled pull request 03#4970: Bump extensions/Score from `932fe9e` to `4e642a6` - 13https://github.com/miraheze/mediawiki/pull/4970 [00:51:25] [02mediawiki] 07dependabot[bot] opened pull request 03#4978: Bump extensions/GlobalBlocking from `2981bfc` to `b7441ba` - 13https://github.com/miraheze/mediawiki/pull/4978 [00:51:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalBlocking-b7441ba - 13https://github.com/miraheze/mediawiki [00:51:28] [02mediawiki] 07github-actions[bot] labeled pull request 03#4973: Bump extensions/Renameuser from `d112548` to `74f89d1` - 13https://github.com/miraheze/mediawiki/pull/4973 [00:51:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AuthorProtect-f295edc [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/6446993cf148 [00:51:31] [02miraheze/mediawiki] 07dependabot[bot] 036446993 - Bump extensions/AuthorProtect from `0741a64` to `f295edc` [00:51:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AuthorProtect-f295edc - 13https://github.com/miraheze/mediawiki [00:51:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#4972: Bump extensions/UniversalLanguageSelector from `d2b84d7` to `b707511` - 13https://github.com/miraheze/mediawiki/pull/4972 [00:51:36] [02mediawiki] 07dependabot[bot] opened pull request 03#4979: Bump extensions/AuthorProtect from `0741a64` to `f295edc` - 13https://github.com/miraheze/mediawiki/pull/4979 [00:51:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#4972: Bump extensions/UniversalLanguageSelector from `d2b84d7` to `b707511` - 13https://github.com/miraheze/mediawiki/pull/4972 [00:51:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#4972: Bump extensions/UniversalLanguageSelector from `d2b84d7` to `b707511` - 13https://github.com/miraheze/mediawiki/pull/4972 [00:51:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-55a7813 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b9891e735e4a [00:51:42] [02miraheze/mediawiki] 07dependabot[bot] 03b9891e7 - Bump extensions/MirahezeMagic from `fb3b068` to `55a7813` [00:51:43] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-55a7813 - 13https://github.com/miraheze/mediawiki [00:51:45] [02mediawiki] 07dependabot[bot] opened pull request 03#4980: Bump extensions/MirahezeMagic from `fb3b068` to `55a7813` - 13https://github.com/miraheze/mediawiki/pull/4980 [00:51:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#4974: Bump extensions/Babel from `5b01abf` to `559a00a` - 13https://github.com/miraheze/mediawiki/pull/4974 [00:51:47] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.71, 9.64, 9.56 [00:51:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Echo-ad5125d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/524a8ff95e2b [00:51:49] [02miraheze/mediawiki] 07dependabot[bot] 03524a8ff - Bump extensions/Echo from `34677a0` to `ad5125d` [00:51:51] [02mediawiki] 07dependabot[bot] opened pull request 03#4981: Bump extensions/Echo from `34677a0` to `ad5125d` - 13https://github.com/miraheze/mediawiki/pull/4981 [00:51:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Echo-ad5125d - 13https://github.com/miraheze/mediawiki [00:51:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Popups-6f30daa [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/fa4226ab08a5 [00:51:55] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:51:55] [02miraheze/mediawiki] 07dependabot[bot] 03fa4226a - Bump extensions/Popups from `669cc18` to `6f30daa` [00:51:57] [02mediawiki] 07dependabot[bot] opened pull request 03#4982: Bump extensions/Popups from `669cc18` to `6f30daa` - 13https://github.com/miraheze/mediawiki/pull/4982 [00:51:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Popups-6f30daa - 13https://github.com/miraheze/mediawiki [00:51:59] PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.81, 9.02, 9.17 [00:52:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TemplateData-13690c5 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/834fbc0e047a [00:52:01] [02miraheze/mediawiki] 07dependabot[bot] 03834fbc0 - Bump extensions/TemplateData from `2896aeb` to `13690c5` [00:52:03] [02mediawiki] 07dependabot[bot] opened pull request 03#4983: Bump extensions/TemplateData from `2896aeb` to `13690c5` - 13https://github.com/miraheze/mediawiki/pull/4983 [00:52:04] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TemplateData-13690c5 - 13https://github.com/miraheze/mediawiki [00:52:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Interwiki-2e5d26e [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e748ea0a5cec [00:52:07] [02miraheze/mediawiki] 07dependabot[bot] 03e748ea0 - Bump extensions/Interwiki from `a525e3e` to `2e5d26e` [00:52:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:52:08] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:09] [02mediawiki] 07dependabot[bot] opened pull request 03#4984: Bump extensions/Interwiki from `a525e3e` to `2e5d26e` - 13https://github.com/miraheze/mediawiki/pull/4984 [00:52:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#4975: Bump extensions/StandardDialogs from `f601fed` to `73b2d03` - 13https://github.com/miraheze/mediawiki/pull/4975 [00:52:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Interwiki-2e5d26e - 13https://github.com/miraheze/mediawiki [00:52:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#4976: Bump extensions/GlobalUsage from `87146f0` to `70afccc` - 13https://github.com/miraheze/mediawiki/pull/4976 [00:52:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CategoryTree-e1620b2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4419bcb8d59b [00:52:16] [02miraheze/mediawiki] 07dependabot[bot] 034419bcb - Bump extensions/CategoryTree from `64c6802` to `e1620b2` [00:52:18] [02mediawiki] 07dependabot[bot] opened pull request 03#4985: Bump extensions/CategoryTree from `64c6802` to `e1620b2` - 13https://github.com/miraheze/mediawiki/pull/4985 [00:52:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CategoryTree-e1620b2 - 13https://github.com/miraheze/mediawiki [00:52:21] [02mediawiki] 07github-actions[bot] labeled pull request 03#4979: Bump extensions/AuthorProtect from `0741a64` to `f295edc` - 13https://github.com/miraheze/mediawiki/pull/4979 [00:52:22] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalNewFiles-e9282be [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/cc4302c70264 [00:52:24] [02miraheze/mediawiki] 07dependabot[bot] 03cc4302c - Bump extensions/GlobalNewFiles from `788cef8` to `e9282be` [00:52:25] [02mediawiki] 07github-actions[bot] labeled pull request 03#4977: Bump extensions/DiscussionTools from `2fcc0eb` to `397361d` - 13https://github.com/miraheze/mediawiki/pull/4977 [00:52:27] [02mediawiki] 07dependabot[bot] opened pull request 03#4986: Bump extensions/GlobalNewFiles from `788cef8` to `e9282be` - 13https://github.com/miraheze/mediawiki/pull/4986 [00:52:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalNewFiles-e9282be - 13https://github.com/miraheze/mediawiki [00:52:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#4978: Bump extensions/GlobalBlocking from `2981bfc` to `b7441ba` - 13https://github.com/miraheze/mediawiki/pull/4978 [00:52:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Cargo-e33a7fd [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2c524a32b0ef [00:52:33] [02miraheze/mediawiki] 07dependabot[bot] 032c524a3 - Bump extensions/Cargo from `b1d6c6e` to `e33a7fd` [00:52:34] [02mediawiki] 07dependabot[bot] opened pull request 03#4987: Bump extensions/Cargo from `b1d6c6e` to `e33a7fd` - 13https://github.com/miraheze/mediawiki/pull/4987 [00:52:36] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Cargo-e33a7fd - 13https://github.com/miraheze/mediawiki [00:52:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#4982: Bump extensions/Popups from `669cc18` to `6f30daa` - 13https://github.com/miraheze/mediawiki/pull/4982 [00:52:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#4981: Bump extensions/Echo from `34677a0` to `ad5125d` - 13https://github.com/miraheze/mediawiki/pull/4981 [00:52:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DynamicPageList-866ecb4 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7ff8d4149e02 [00:52:42] [02miraheze/mediawiki] 07dependabot[bot] 037ff8d41 - Bump extensions/DynamicPageList from `700a600` to `866ecb4` [00:52:43] [02mediawiki] 07dependabot[bot] opened pull request 03#4988: Bump extensions/DynamicPageList from `700a600` to `866ecb4` - 13https://github.com/miraheze/mediawiki/pull/4988 [00:52:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DynamicPageList-866ecb4 - 13https://github.com/miraheze/mediawiki [00:52:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#4980: Bump extensions/MirahezeMagic from `fb3b068` to `55a7813` - 13https://github.com/miraheze/mediawiki/pull/4980 [00:52:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/LockAuthor-8926bff [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4d7fa4c104c2 [00:52:49] [02miraheze/mediawiki] 07dependabot[bot] 034d7fa4c - Bump extensions/LockAuthor from `878b2e6` to `8926bff` [00:52:50] PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:51] [02mediawiki] 07dependabot[bot] opened pull request 03#4989: Bump extensions/LockAuthor from `878b2e6` to `8926bff` - 13https://github.com/miraheze/mediawiki/pull/4989 [00:52:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/LockAuthor-8926bff - 13https://github.com/miraheze/mediawiki [00:52:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#4984: Bump extensions/Interwiki from `a525e3e` to `2e5d26e` - 13https://github.com/miraheze/mediawiki/pull/4984 [00:52:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/OAuth-75168c1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b28682ebfa44 [00:52:56] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:52:57] [02miraheze/mediawiki] 07dependabot[bot] 03b28682e - Bump extensions/OAuth from `5e71c47` to `75168c1` [00:52:58] [02mediawiki] 07dependabot[bot] opened pull request 03#4990: Bump extensions/OAuth from `5e71c47` to `75168c1` - 13https://github.com/miraheze/mediawiki/pull/4990 [00:52:59] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.38, 9.60, 9.97 [00:53:00] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/OAuth-75168c1 - 13https://github.com/miraheze/mediawiki [00:53:01] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Moderation-d8b997e [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/00f36451d3e2 [00:53:03] [02miraheze/mediawiki] 07dependabot[bot] 0300f3645 - Bump extensions/Moderation from `041c357` to `d8b997e` [00:53:04] [02mediawiki] 07dependabot[bot] opened pull request 03#4991: Bump extensions/Moderation from `041c357` to `d8b997e` - 13https://github.com/miraheze/mediawiki/pull/4991 [00:53:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#4983: Bump extensions/TemplateData from `2896aeb` to `13690c5` - 13https://github.com/miraheze/mediawiki/pull/4983 [00:53:07] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Moderation-d8b997e - 13https://github.com/miraheze/mediawiki [00:53:09] [02mediawiki] 07dependabot[bot] opened pull request 03#4992: Bump extensions/ArticlePlaceholder from `23d9495` to `6700354` - 13https://github.com/miraheze/mediawiki/pull/4992 [00:53:10] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ArticlePlaceholder-6700354 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2a9897cf9e21 [00:53:11] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to scsvg [00:53:12] [02miraheze/mediawiki] 07dependabot[bot] 032a9897c - Bump extensions/ArticlePlaceholder from `23d9495` to `6700354` [00:53:13] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb [00:53:13] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ArticlePlaceholder-6700354 - 13https://github.com/miraheze/mediawiki [00:53:14] PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:53:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CreateWiki-2b0f27d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/750d8b1f45ed [00:53:16] [02miraheze/mediawiki] 07dependabot[bot] 03750d8b1 - Bump extensions/CreateWiki from `5698d35` to `2b0f27d` [00:53:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CreateWiki-2b0f27d - 13https://github.com/miraheze/mediawiki [00:53:19] [02mediawiki] 07dependabot[bot] opened pull request 03#4993: Bump extensions/CreateWiki from `5698d35` to `2b0f27d` - 13https://github.com/miraheze/mediawiki/pull/4993 [00:53:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:53:21] [02mediawiki] 07github-actions[bot] labeled pull request 03#4985: Bump extensions/CategoryTree from `64c6802` to `e1620b2` - 13https://github.com/miraheze/mediawiki/pull/4985 [00:53:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#4986: Bump extensions/GlobalNewFiles from `788cef8` to `e9282be` - 13https://github.com/miraheze/mediawiki/pull/4986 [00:53:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikimediaIncubator-2f1e105 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/63f39b6ff4e4 [00:53:24] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.86, 2.82, 3.23 [00:53:25] [02miraheze/mediawiki] 07dependabot[bot] 0363f39b6 - Bump extensions/WikimediaIncubator from `33d1ba9` to `2f1e105` [00:53:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikimediaIncubator-2f1e105 - 13https://github.com/miraheze/mediawiki [00:53:28] [02mediawiki] 07dependabot[bot] opened pull request 03#4994: Bump extensions/WikimediaIncubator from `33d1ba9` to `2f1e105` - 13https://github.com/miraheze/mediawiki/pull/4994 [00:53:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/InputBox-e0a1c32 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7bb0f26316bd [00:53:31] [02miraheze/mediawiki] 07dependabot[bot] 037bb0f26 - Bump extensions/InputBox from `6c2d506` to `e0a1c32` [00:53:33] [02mediawiki] 07dependabot[bot] opened pull request 03#4995: Bump extensions/InputBox from `6c2d506` to `e0a1c32` - 13https://github.com/miraheze/mediawiki/pull/4995 [00:53:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/InputBox-e0a1c32 - 13https://github.com/miraheze/mediawiki [00:53:36] [02mediawiki] 07github-actions[bot] labeled pull request 03#4987: Bump extensions/Cargo from `b1d6c6e` to `e33a7fd` - 13https://github.com/miraheze/mediawiki/pull/4987 [00:53:37] PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:53:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#4987: Bump extensions/Cargo from `b1d6c6e` to `e33a7fd` - 13https://github.com/miraheze/mediawiki/pull/4987 [00:53:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#4988: Bump extensions/DynamicPageList from `700a600` to `866ecb4` - 13https://github.com/miraheze/mediawiki/pull/4988 [00:53:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#4987: Bump extensions/Cargo from `b1d6c6e` to `e33a7fd` - 13https://github.com/miraheze/mediawiki/pull/4987 [00:53:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ConfirmEdit-95640eb [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0bb632814b12 [00:53:43] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.752 second response time [00:53:43] [02miraheze/mediawiki] 07dependabot[bot] 030bb6328 - Bump extensions/ConfirmEdit from `a591b21` to `95640eb` [00:53:45] [02mediawiki] 07dependabot[bot] opened pull request 03#4996: Bump extensions/ConfirmEdit from `a591b21` to `95640eb` - 13https://github.com/miraheze/mediawiki/pull/4996 [00:53:46] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ConfirmEdit-95640eb - 13https://github.com/miraheze/mediawiki [00:53:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#4991: Bump extensions/Moderation from `041c357` to `d8b997e` - 13https://github.com/miraheze/mediawiki/pull/4991 [00:53:49] PROBLEM - test101 Check Gluster Clients on test101 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [00:53:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#4989: Bump extensions/LockAuthor from `878b2e6` to `8926bff` - 13https://github.com/miraheze/mediawiki/pull/4989 [00:53:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageTriage-0e564dd [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5cf6920baa35 [00:53:52] [02miraheze/mediawiki] 07dependabot[bot] 035cf6920 - Bump extensions/PageTriage from `d9440b4` to `0e564dd` [00:53:53] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to scsvg - SUCCESS in 41s [00:53:54] [02mediawiki] 07dependabot[bot] opened pull request 03#4997: Bump extensions/PageTriage from `d9440b4` to `0e564dd` - 13https://github.com/miraheze/mediawiki/pull/4997 [00:53:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:53:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageTriage-0e564dd - 13https://github.com/miraheze/mediawiki [00:53:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#4990: Bump extensions/OAuth from `5e71c47` to `75168c1` - 13https://github.com/miraheze/mediawiki/pull/4990 [00:53:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MobileFrontend-82da625 - 13https://github.com/miraheze/mediawiki [00:54:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MobileFrontend-82da625 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e596b8675421 [00:54:01] [02miraheze/mediawiki] 07dependabot[bot] 03e596b86 - Bump extensions/MobileFrontend from `f83e725` to `82da625` [00:54:02] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:54:03] [02mediawiki] 07dependabot[bot] opened pull request 03#4998: Bump extensions/MobileFrontend from `f83e725` to `82da625` - 13https://github.com/miraheze/mediawiki/pull/4998 [00:54:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#4993: Bump extensions/CreateWiki from `5698d35` to `2b0f27d` - 13https://github.com/miraheze/mediawiki/pull/4993 [00:54:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#4993: Bump extensions/CreateWiki from `5698d35` to `2b0f27d` - 13https://github.com/miraheze/mediawiki/pull/4993 [00:54:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#4992: Bump extensions/ArticlePlaceholder from `23d9495` to `6700354` - 13https://github.com/miraheze/mediawiki/pull/4992 [00:54:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MatomoAnalytics-7bb72ad [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ed46d1a97d03 [00:54:10] [02miraheze/mediawiki] 07dependabot[bot] 03ed46d1a - Bump extensions/MatomoAnalytics from `ba5a4c8` to `7bb72ad` [00:54:12] [02mediawiki] 07dependabot[bot] opened pull request 03#4999: Bump extensions/MatomoAnalytics from `ba5a4c8` to `7bb72ad` - 13https://github.com/miraheze/mediawiki/pull/4999 [00:54:13] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.56, 9.38, 9.01 [00:54:13] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MatomoAnalytics-7bb72ad - 13https://github.com/miraheze/mediawiki [00:54:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Gadgets-f958b46 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/767be2f9d2db [00:54:16] [02miraheze/mediawiki] 07dependabot[bot] 03767be2f - Bump extensions/Gadgets from `4979fe1` to `f958b46` [00:54:18] [02mediawiki] 07dependabot[bot] opened pull request 03#5000: Bump extensions/Gadgets from `4979fe1` to `f958b46` - 13https://github.com/miraheze/mediawiki/pull/5000 [00:54:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Gadgets-f958b46 - 13https://github.com/miraheze/mediawiki [00:54:20] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 4.083 second response time [00:54:21] [02mediawiki] 07github-actions[bot] labeled pull request 03#4995: Bump extensions/InputBox from `6c2d506` to `e0a1c32` - 13https://github.com/miraheze/mediawiki/pull/4995 [00:54:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#4994: Bump extensions/WikimediaIncubator from `33d1ba9` to `2f1e105` - 13https://github.com/miraheze/mediawiki/pull/4994 [00:54:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Wikibase-5007351 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c4c5100aa4c0 [00:54:25] [02miraheze/mediawiki] 07dependabot[bot] 03c4c5100 - Bump extensions/Wikibase from `2261946` to `5007351` [00:54:27] [02mediawiki] 07dependabot[bot] opened pull request 03#5001: Bump extensions/Wikibase from `2261946` to `5007351` - 13https://github.com/miraheze/mediawiki/pull/5001 [00:54:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Wikibase-5007351 - 13https://github.com/miraheze/mediawiki [00:54:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Evelution-d492062 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2d71677d5b72 [00:54:32] [02miraheze/mediawiki] 07dependabot[bot] 032d71677 - Bump skins/Evelution from `833020a` to `d492062` [00:54:33] [02mediawiki] 07dependabot[bot] opened pull request 03#5002: Bump skins/Evelution from `833020a` to `d492062` - 13https://github.com/miraheze/mediawiki/pull/5002 [00:54:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:54:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Evelution-d492062 - 13https://github.com/miraheze/mediawiki [00:54:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DisplayTitle-f3a4bf4 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/626e82707445 [00:54:37] [02miraheze/mediawiki] 07dependabot[bot] 03626e827 - Bump extensions/DisplayTitle from `d480066` to `f3a4bf4` [00:54:39] [02mediawiki] 07dependabot[bot] opened pull request 03#5003: Bump extensions/DisplayTitle from `d480066` to `f3a4bf4` - 13https://github.com/miraheze/mediawiki/pull/5003 [00:54:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DisplayTitle-f3a4bf4 - 13https://github.com/miraheze/mediawiki [00:54:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#4997: Bump extensions/PageTriage from `d9440b4` to `0e564dd` - 13https://github.com/miraheze/mediawiki/pull/4997 [00:54:43] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/BlogPage-69a93d7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a9e6dedc3495 [00:54:45] [02miraheze/mediawiki] 07dependabot[bot] 03a9e6ded - Bump extensions/BlogPage from `c884723` to `69a93d7` [00:54:46] [02mediawiki] 07dependabot[bot] opened pull request 03#5004: Bump extensions/BlogPage from `c884723` to `69a93d7` - 13https://github.com/miraheze/mediawiki/pull/5004 [00:54:48] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/BlogPage-69a93d7 - 13https://github.com/miraheze/mediawiki [00:54:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#4996: Bump extensions/ConfirmEdit from `a591b21` to `95640eb` - 13https://github.com/miraheze/mediawiki/pull/4996 [00:54:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CentralNotice-e206d09 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/98f4a8f1159a [00:54:52] [02miraheze/mediawiki] 07dependabot[bot] 0398f4a8f - Bump extensions/CentralNotice from `794de94` to `e206d09` [00:54:54] [02mediawiki] 07dependabot[bot] opened pull request 03#5005: Bump extensions/CentralNotice from `794de94` to `e206d09` - 13https://github.com/miraheze/mediawiki/pull/5005 [00:54:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CentralNotice-e206d09 - 13https://github.com/miraheze/mediawiki [00:54:56] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.28, 10.25, 9.80 [00:54:57] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Translate-12e795e [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c480f28b55f4 [00:54:59] [02miraheze/mediawiki] 07dependabot[bot] 03c480f28 - Bump extensions/Translate from `266785e` to `12e795e` [00:54:59] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.976 second response time [00:55:00] [02mediawiki] 07dependabot[bot] opened pull request 03#5006: Bump extensions/Translate from `266785e` to `12e795e` - 13https://github.com/miraheze/mediawiki/pull/5006 [00:55:01] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Translate-12e795e - 13https://github.com/miraheze/mediawiki [00:55:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#4999: Bump extensions/MatomoAnalytics from `ba5a4c8` to `7bb72ad` - 13https://github.com/miraheze/mediawiki/pull/4999 [00:55:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#4999: Bump extensions/MatomoAnalytics from `ba5a4c8` to `7bb72ad` - 13https://github.com/miraheze/mediawiki/pull/4999 [00:55:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#4998: Bump extensions/MobileFrontend from `f83e725` to `82da625` - 13https://github.com/miraheze/mediawiki/pull/4998 [00:55:08] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/BlueSky-c96dbc8 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5cf385415fd4 [00:55:09] [02miraheze/mediawiki] 07dependabot[bot] 035cf3854 - Bump skins/BlueSky from `ff3d020` to `c96dbc8` [00:55:10] [02mediawiki] 07dependabot[bot] opened pull request 03#5007: Bump skins/BlueSky from `ff3d020` to `c96dbc8` - 13https://github.com/miraheze/mediawiki/pull/5007 [00:55:11] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.239 second response time [00:55:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/BlueSky-c96dbc8 - 13https://github.com/miraheze/mediawiki [00:55:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#5000: Bump extensions/Gadgets from `4979fe1` to `f958b46` - 13https://github.com/miraheze/mediawiki/pull/5000 [00:55:15] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.327 second response time [00:55:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MultimediaViewer-fae4afc [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/eee0b16aba43 [00:55:16] [02miraheze/mediawiki] 07dependabot[bot] 03eee0b16 - Bump extensions/MultimediaViewer from `6b87177` to `fae4afc` [00:55:18] [02mediawiki] 07dependabot[bot] opened pull request 03#5008: Bump extensions/MultimediaViewer from `6b87177` to `fae4afc` - 13https://github.com/miraheze/mediawiki/pull/5008 [00:55:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MultimediaViewer-fae4afc - 13https://github.com/miraheze/mediawiki [00:55:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Vector-5dea00c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0a38e179fb28 [00:55:22] [02miraheze/mediawiki] 07dependabot[bot] 030a38e17 - Bump skins/Vector from `e826214` to `5dea00c` [00:55:24] [02mediawiki] 07dependabot[bot] opened pull request 03#5009: Bump skins/Vector from `e826214` to `5dea00c` - 13https://github.com/miraheze/mediawiki/pull/5009 [00:55:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Vector-5dea00c - 13https://github.com/miraheze/mediawiki [00:55:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DarkMode-ca482df [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/42c6b93fe9a6 [00:55:29] [02miraheze/mediawiki] 07dependabot[bot] 0342c6b93 - Bump extensions/DarkMode from `8adedfd` to `ca482df` [00:55:30] [02mediawiki] 07dependabot[bot] opened pull request 03#5010: Bump extensions/DarkMode from `8adedfd` to `ca482df` - 13https://github.com/miraheze/mediawiki/pull/5010 [00:55:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DarkMode-ca482df - 13https://github.com/miraheze/mediawiki [00:55:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#5004: Bump extensions/BlogPage from `c884723` to `69a93d7` - 13https://github.com/miraheze/mediawiki/pull/5004 [00:55:35] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Timeline-d2082b6 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5d7dcbf83542 [00:55:36] [02miraheze/mediawiki] 07dependabot[bot] 035d7dcbf - Bump extensions/Timeline from `93542d8` to `d2082b6` [00:55:37] [02mediawiki] 07dependabot[bot] opened pull request 03#5011: Bump extensions/Timeline from `93542d8` to `d2082b6` - 13https://github.com/miraheze/mediawiki/pull/5011 [00:55:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#5003: Bump extensions/DisplayTitle from `d480066` to `f3a4bf4` - 13https://github.com/miraheze/mediawiki/pull/5003 [00:55:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Timeline-d2082b6 - 13https://github.com/miraheze/mediawiki [00:55:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#5003: Bump extensions/DisplayTitle from `d480066` to `f3a4bf4` - 13https://github.com/miraheze/mediawiki/pull/5003 [00:55:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Linter-71d2b3c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/66d6de7f8db5 [00:55:45] [02miraheze/mediawiki] 07dependabot[bot] 0366d6de7 - Bump extensions/Linter from `ca03a0d` to `71d2b3c` [00:55:47] [02mediawiki] 07dependabot[bot] opened pull request 03#5012: Bump extensions/Linter from `ca03a0d` to `71d2b3c` - 13https://github.com/miraheze/mediawiki/pull/5012 [00:55:48] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Linter-71d2b3c - 13https://github.com/miraheze/mediawiki [00:55:50] [02mediawiki] 07github-actions[bot] labeled pull request 03#5002: Bump skins/Evelution from `833020a` to `d492062` - 13https://github.com/miraheze/mediawiki/pull/5002 [00:55:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#5002: Bump skins/Evelution from `833020a` to `d492062` - 13https://github.com/miraheze/mediawiki/pull/5002 [00:55:53] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/3D-caf3f10 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/33df608c4e94 [00:55:54] [02miraheze/mediawiki] 07dependabot[bot] 0333df608 - Bump extensions/3D from `0610ca8` to `caf3f10` [00:55:56] [02mediawiki] 07dependabot[bot] opened pull request 03#5013: Bump extensions/3D from `0610ca8` to `caf3f10` - 13https://github.com/miraheze/mediawiki/pull/5013 [00:55:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/3D-caf3f10 - 13https://github.com/miraheze/mediawiki [00:55:58] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.911 second response time [00:55:59] [02mediawiki] 07github-actions[bot] labeled pull request 03#5001: Bump extensions/Wikibase from `2261946` to `5007351` - 13https://github.com/miraheze/mediawiki/pull/5001 [00:56:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#5007: Bump skins/BlueSky from `ff3d020` to `c96dbc8` - 13https://github.com/miraheze/mediawiki/pull/5007 [00:56:02] [02mediawiki] 07github-actions[bot] labeled pull request 03#5005: Bump extensions/CentralNotice from `794de94` to `e206d09` - 13https://github.com/miraheze/mediawiki/pull/5005 [00:56:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Medik-1840eb1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/66d8b9976b43 [00:56:05] [02miraheze/mediawiki] 07dependabot[bot] 0366d8b99 - Bump skins/Medik from `833353a` to `1840eb1` [00:56:06] [02mediawiki] 07dependabot[bot] opened pull request 03#5014: Bump skins/Medik from `833353a` to `1840eb1` - 13https://github.com/miraheze/mediawiki/pull/5014 [00:56:08] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Medik-1840eb1 - 13https://github.com/miraheze/mediawiki [00:56:09] [02mediawiki] 07github-actions[bot] labeled pull request 03#5009: Bump skins/Vector from `e826214` to `5dea00c` - 13https://github.com/miraheze/mediawiki/pull/5009 [00:56:11] [02mediawiki] 07github-actions[bot] labeled pull request 03#5010: Bump extensions/DarkMode from `8adedfd` to `ca482df` - 13https://github.com/miraheze/mediawiki/pull/5010 [00:56:11] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:56:12] [02mediawiki] 07github-actions[bot] labeled pull request 03#5008: Bump extensions/MultimediaViewer from `6b87177` to `fae4afc` - 13https://github.com/miraheze/mediawiki/pull/5008 [00:56:14] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/MinervaNeue-9b6acac [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2c6ed7ac7b48 [00:56:15] [02miraheze/mediawiki] 07dependabot[bot] 032c6ed7a - Bump skins/MinervaNeue from `15047cd` to `9b6acac` [00:56:17] [02mediawiki] 07dependabot[bot] opened pull request 03#5015: Bump skins/MinervaNeue from `15047cd` to `9b6acac` - 13https://github.com/miraheze/mediawiki/pull/5015 [00:56:17] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.394 second response time [00:56:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/MinervaNeue-9b6acac - 13https://github.com/miraheze/mediawiki [00:56:20] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageForms-86d9007 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/bdeb5fd95d45 [00:56:21] [02miraheze/mediawiki] 07dependabot[bot] 03bdeb5fd - Bump extensions/PageForms from `3df1b78` to `86d9007` [00:56:23] [02mediawiki] 07dependabot[bot] opened pull request 03#5016: Bump extensions/PageForms from `3df1b78` to `86d9007` - 13https://github.com/miraheze/mediawiki/pull/5016 [00:56:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageForms-86d9007 - 13https://github.com/miraheze/mediawiki [00:56:26] [02mediawiki] 07github-actions[bot] labeled pull request 03#5006: Bump extensions/Translate from `266785e` to `12e795e` - 13https://github.com/miraheze/mediawiki/pull/5006 [00:56:27] [02mediawiki] 07github-actions[bot] labeled pull request 03#5006: Bump extensions/Translate from `266785e` to `12e795e` - 13https://github.com/miraheze/mediawiki/pull/5006 [00:56:29] [02mediawiki] 07github-actions[bot] labeled pull request 03#5006: Bump extensions/Translate from `266785e` to `12e795e` - 13https://github.com/miraheze/mediawiki/pull/5006 [00:56:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#5011: Bump extensions/Timeline from `93542d8` to `d2082b6` - 13https://github.com/miraheze/mediawiki/pull/5011 [00:56:32] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CentralAuth-14aebc2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9a6f8b9ba268 [00:56:33] [02miraheze/mediawiki] 07dependabot[bot] 039a6f8b9 - Bump extensions/CentralAuth from `80029dd` to `14aebc2` [00:56:35] [02mediawiki] 07dependabot[bot] opened pull request 03#5017: Bump extensions/CentralAuth from `80029dd` to `14aebc2` - 13https://github.com/miraheze/mediawiki/pull/5017 [00:56:36] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CentralAuth-14aebc2 - 13https://github.com/miraheze/mediawiki [00:56:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ContactPage-93275fe [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/cdf4bcac4f4f [00:56:39] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.753 second response time [00:56:39] [02miraheze/mediawiki] 07dependabot[bot] 03cdf4bca - Bump extensions/ContactPage from `fbd203c` to `93275fe` [00:56:41] [02mediawiki] 07dependabot[bot] opened pull request 03#5018: Bump extensions/ContactPage from `fbd203c` to `93275fe` - 13https://github.com/miraheze/mediawiki/pull/5018 [00:56:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ContactPage-93275fe - 13https://github.com/miraheze/mediawiki [00:56:44] [02mediawiki] 07github-actions[bot] labeled pull request 03#5012: Bump extensions/Linter from `ca03a0d` to `71d2b3c` - 13https://github.com/miraheze/mediawiki/pull/5012 [00:56:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Anisa-332e5ed [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7512da8e3dcf [00:56:47] [02miraheze/mediawiki] 07dependabot[bot] 037512da8 - Bump skins/Anisa from `46d144b` to `332e5ed` [00:56:48] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Anisa-332e5ed - 13https://github.com/miraheze/mediawiki [00:56:50] [02mediawiki] 07dependabot[bot] opened pull request 03#5019: Bump skins/Anisa from `46d144b` to `332e5ed` - 13https://github.com/miraheze/mediawiki/pull/5019 [00:56:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#5013: Bump extensions/3D from `0610ca8` to `caf3f10` - 13https://github.com/miraheze/mediawiki/pull/5013 [00:56:53] [02mediawiki] 07github-actions[bot] labeled pull request 03#5014: Bump skins/Medik from `833353a` to `1840eb1` - 13https://github.com/miraheze/mediawiki/pull/5014 [00:56:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TimedMediaHandler-c92b7bc [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7c3e6c3a186d [00:56:56] [02miraheze/mediawiki] 07dependabot[bot] 037c3e6c3 - Bump extensions/TimedMediaHandler from `16976ec` to `c92b7bc` [00:56:57] [02mediawiki] 07dependabot[bot] opened pull request 03#5020: Bump extensions/TimedMediaHandler from `16976ec` to `c92b7bc` - 13https://github.com/miraheze/mediawiki/pull/5020 [00:56:59] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TimedMediaHandler-c92b7bc - 13https://github.com/miraheze/mediawiki [00:57:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#5016: Bump extensions/PageForms from `3df1b78` to `86d9007` - 13https://github.com/miraheze/mediawiki/pull/5016 [00:57:01] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.153 second response time [00:57:02] [02mediawiki] 07github-actions[bot] labeled pull request 03#5015: Bump skins/MinervaNeue from `15047cd` to `9b6acac` - 13https://github.com/miraheze/mediawiki/pull/5015 [00:57:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#5017: Bump extensions/CentralAuth from `80029dd` to `14aebc2` - 13https://github.com/miraheze/mediawiki/pull/5017 [00:57:05] [02mediawiki] 07github-actions[bot] labeled pull request 03#5018: Bump extensions/ContactPage from `fbd203c` to `93275fe` - 13https://github.com/miraheze/mediawiki/pull/5018 [00:57:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#5019: Bump skins/Anisa from `46d144b` to `332e5ed` - 13https://github.com/miraheze/mediawiki/pull/5019 [00:57:08] [02mediawiki] 07github-actions[bot] labeled pull request 03#5020: Bump extensions/TimedMediaHandler from `16976ec` to `c92b7bc` - 13https://github.com/miraheze/mediawiki/pull/5020 [00:57:08] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.20, 9.34, 9.07 [00:57:49] !log universalomega@mwtask111:/srv/mediawiki-staging/config/.git/objects$ sudo chown -R www-data:www-data * [00:58:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.02, 9.87, 9.44 [00:58:15] !log reboot test101 [00:58:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:59:12] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.10, 10.58, 10.18 [00:59:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:00:19] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.39, 10.30, 9.49 [01:01:47] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:02:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:03:21] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.28, 9.38, 9.27 [01:03:32] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [01:04:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.34, 8.66, 9.45 [01:04:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.20, 9.56, 9.52 [01:06:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:07:15] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.30, 9.95, 9.80 [01:07:37] CosmicAlpha, whose running maintenance script? [01:07:59] whoever didn't specify -u "Miraheze maintenance script" so their edits would be flood flagged [01:08:04] nevertheless I can fix this [01:08:09] with StreamBotMH [01:08:24] the whole RC is backlogged [01:08:48] dmehus: I am running a rebuildImages script, but there is not that option for it. that is all I am running though. [01:09:16] CosmicAlpha, there should be, Reception123 uses it when importing images [01:09:26] I'll see if I can use StreamBotMh [01:09:26] i am rebuilding not importing [01:09:46] to fix the missing images on commons wiki [01:10:49] Ergh [01:10:51] not working [01:10:58] since StreamBot-read is in -feed [01:11:05] Can you pause it please, CosmicAlpha? [01:11:30] oh wait [01:11:31] I'm guessing pausing wouldn't do anything at this point [01:11:37] maybe I can unwatch commonswiki [01:13:42] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.75, 9.20, 9.64 [01:13:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.44, 9.74, 9.60 [01:14:07] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:14:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.52, 4.45, 3.90 [01:15:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.23, 4.45, 3.66 [01:16:38] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildImages.php --wiki=commonswiki --missing (END - exit=0) [01:17:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:20:07] dmehus: it finished now... [01:20:26] sorry about that but I do not think there is an option to specify username for that. [01:20:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.94, 3.89 [01:20:49] CosmicAlpha, oh ok, maybe we could submit an upstream task to add a username parameter to rebuildImages.php? [01:21:13] I think that would be quite useful [01:21:14] it is possible it has it if it's generic and not listed on documentation. I will look. [01:21:23] cool, thanks :) [01:23:13] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.68, 9.53, 9.51 [01:23:25] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.16, 3.72, 3.79 [01:23:26] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:24:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8e5e4c3273c0...c2e52e05bf4a [01:24:11] [02miraheze/mw-config] 07Universal-Omega 03c2e52e0 - fix [01:24:19] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to scsvg [01:24:28] dmehus: no problem [01:24:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:40] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to scsvg - SUCCESS in 20s [01:24:45] sorry about the flood, I did not know this would log at all anyway. [01:24:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:53] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.38, 3.76, 3.83 [01:25:14] miraheze/mw-config - Universal-Omega the build passed. [01:25:43] CosmicAlpha, yeah, no problem, me either. We don't run this script often, I think [01:26:04] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.50, 9.13, 9.36 [01:26:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.84, 3.18, 3.57 [01:26:30] dmehus: yeah I think it is the first time I have ran it. [01:26:47] CosmicAlpha, oh wow [01:26:47] now 2 issues from migration should be fixed, a few to go... [01:26:54] yeah, that's good [01:27:09] any idea what caused the issue with the images? [01:27:21] database migration [01:27:25] oh [01:27:40] did the paths to the URLs change? [01:27:48] no [01:27:49] like is it not static.miraheze.org or something? [01:27:51] oh [01:28:05] they just existed in file system, just not in images/oldimages tables [01:28:11] oh [01:28:29] only commonswiki it seems also [01:28:35] from what I noticed [01:28:44] I wonder why that table data did not migrate 🤔 [01:28:58] CosmicAlpha, ah, maybe it's related to GlobalUsage? [01:29:07] since Commons is the main wiki for that? [01:29:10] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.93, 2.75, 3.35 [01:33:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 3.53, 3.58 [01:34:00] The "DNS misconfiguration" on robloxapi.wiki should be fixedf! [01:34:00] https://cdn.discordapp.com/attachments/808001911868489748/932447632985579580/unknown.png [01:34:25] 😔 No more cloudflare [01:34:46] dmehus: could be [01:35:21] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.73, 9.58, 9.91 [01:36:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.20, 3.43 [01:38:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.75, 9.64, 9.85 [01:38:38] Agent: RC feed delay is running 19 minutes behind [01:38:49] so should be caught up in about 25 minutes or so [01:39:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.24, 2.84, 3.26 [01:47:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.32, 3.73, 3.54 [01:47:28] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.78, 9.64, 9.46 [01:47:39] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [01:47:42] !log [@test101] starting deploy of {'config': True} to skip [01:47:43] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:48:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:48:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:49:29] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [01:50:58] Agent, CosmicAlpha: RC feeds all caught up [01:53:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.72, 4.25, 3.84 [01:54:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.35, 5.21, 4.08 [01:55:11] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.27, 4.40, 3.76 [01:55:49] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:57:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.79, 8.67, 9.27 [02:00:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.81, 3.02, 3.49 [02:00:16] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.04, 3.32, 3.56 [02:00:16] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:00:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.98, 9.20, 9.69 [02:01:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.55, 3.61, 3.60 [02:03:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.58, 2.91, 3.36 [02:03:13] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [02:03:42] RECOVERY - robloxapi.wiki - reverse DNS on sslhost is OK: SSL OK - robloxapi.wiki reverse DNS resolves to cp20.miraheze.org - CNAME FLAT [02:05:08] PROBLEM - robloxapi.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'robloxapi.wiki' expired on Sun 28 Nov 2021 10:26:05 GMT +0000. [02:06:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:06:17] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.54, 2.87, 3.30 [02:12:18] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.16, 10.11, 9.77 [02:12:54] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.01, 3.05, 3.28 [02:12:58] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.34, 9.26, 8.91 [02:15:05] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.93, 9.30, 9.17 [02:15:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.02, 9.24, 9.47 [02:16:21] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:18:31] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.57, 9.98, 9.73 [02:19:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c2e52e05bf4a...dfa6d0cdc79c [02:19:09] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.28, 9.61, 9.26 [02:19:10] [02miraheze/mw-config] 07Universal-Omega 03dfa6d0c - set $wgJobRunRate = 0.1; [02:19:29] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to scsvg [02:19:55] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to scsvg - SUCCESS in 26s [02:20:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:20:19] miraheze/mw-config - Universal-Omega the build passed. [02:20:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:21:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.15, 9.42, 9.31 [02:21:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.73, 9.73, 9.71 [02:23:32] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [02:30:02] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:31:03] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.54, 9.57, 9.38 [02:31:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.55, 3.22 [02:33:12] CosmicAlpha: the CentralAuth auto-sign-in is taking like a couple minutes on wikis with custom domains [02:33:22] not sure why, but likely related to the performance issues? [02:33:46] dmehus: it's on my list to get to [02:33:55] CosmicAlpha: ack, awesome, and ty :) [02:34:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.87, 9.71, 9.46 [02:34:20] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.21, 10.51, 10.04 [02:34:31] I just deployed a cache change to mw101 and mw111 and now meta main page loaded in 52ms! [02:34:55] dmehus: no problem [02:34:57] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.89, 3.35, 3.21 [02:35:34] that is some awesome load times! [02:35:36] CosmicAlpha, wow, cool :) [02:35:43] so it does seem to be cache related [02:36:10] yeah I always new cache could speed up load, but the server load and php-fpm issues is unrelated I think. [02:37:00] !log [universalomega@mwtask111] starting deploy of {'config': True} to mw101,mw111 [02:37:03] !log [universalomega@mwtask111] finished deploy of {'config': True} to mw101,mw111 - SUCCESS in 2s [02:37:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.81, 9.32, 9.64 [02:37:37] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.52, 9.64, 9.35 [02:37:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:42] !log [universalomega@mwtask111] starting deploy of {'config': True} to mw102,mw112 [02:37:53] !log [universalomega@mwtask111] finished deploy of {'config': True} to mw102,mw112 - SUCCESS in 11s [02:37:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:38:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:38:26] rc just loaded in 500ms on meta! [02:39:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.70, 3.49, 3.41 [02:40:07] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.16, 10.40, 9.99 [02:40:46] every since I have been here I have NEVER seen them that low! [02:41:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.87, 8.73, 9.92 [02:41:35] !log [universalomega@mwtask111] starting deploy of {'config': True} to mw121,mw122 [02:41:45] !log [universalomega@mwtask111] finished deploy of {'config': True} to mw121,mw122 - SUCCESS in 9s [02:41:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:41:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.21, 2.71, 3.10 [02:41:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:43:02] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.58, 8.86, 9.09 [02:43:13] [02mw-config] 07Universal-Omega edited pull request 03#4393: Redo caching configuration - 13https://github.com/miraheze/mw-config/pull/4393 [02:43:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.21, 8.82, 9.75 [02:44:29] dmehus: do you notice it on meta now? [02:45:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.58, 9.31, 9.72 [02:46:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.82, 9.30, 9.73 [02:47:20] RECOVERY - test101 Check Gluster Clients on test101 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [02:47:25] !log mount /mnt/mediawiki-static on test101 [02:47:28] !log [@test101] starting deploy of {'config': True} to skip [02:47:29] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [02:47:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:47:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:47:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:49:00] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.76, 9.92, 9.91 [02:51:46] !log [@mwtask111] starting deploy of {'config': True} to scsvg [02:51:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:51:53] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 6s [02:51:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:10] [02mw-config] 07Universal-Omega closed pull request 03#4393: Redo caching configuration - 13https://github.com/miraheze/mw-config/pull/4393 [02:52:12] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [02:52:13] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [02:52:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/dfa6d0cdc79c...da50161a3434 [02:52:16] [02miraheze/mw-config] 07Universal-Omega 03da50161 - Redo caching configuration (#4393) [02:52:32] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to scsvg [02:52:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:47] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to scsvg - SUCCESS in 14s [02:52:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:53:11] miraheze/mw-config - Universal-Omega the build passed. [02:53:12] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to skip [02:53:12] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to skip - SUCCESS in 1s [02:53:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:53:20] CosmicAlpha, it does seem to be a bit faster, let me pay closer attention now [02:53:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:53:57] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.59, 9.37, 9.04 [02:54:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.51, 9.04, 9.62 [02:54:46] CosmicAlpha, interesting commit re: https://github.com/miraheze/mw-config/compare/dfa6d0cdc79c...da50161a3434 [02:54:46] [url] Comparing dfa6d0cdc79c...da50161a3434 · miraheze/mw-config · GitHub | github.com [02:55:03] but yeah I have seen load times at like 500 ms, but it was only a short time last summer [02:55:06] not since then [02:55:14] and very rarely before that [02:55:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.26, 9.64, 9.89 [02:55:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.80, 9.20, 9.93 [02:56:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/da50161a3434...6769d239f766 [02:56:10] [02miraheze/mw-config] 07Universal-Omega 036769d23 - lower $wgJobRunRate to 0.04 [02:56:21] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to scsvg [02:56:21] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:56:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:56:26] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to skip [02:56:27] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to skip - SUCCESS in 1s [02:56:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:56:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:56:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.19, 8.28, 8.67 [02:56:46] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to scsvg - SUCCESS in 25s [02:56:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:57:08] miraheze/mw-config - Universal-Omega the build passed. [02:57:18] CosmicAlpha, much better, down to 1250ms [02:57:24] I can live with that even [02:57:36] but like 3000-10000ms is not good [02:57:53] dmehus: yeah! I am happy with this also, main page loads in 50ms-500ms for me now! [02:58:00] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.38, 9.89, 9.94 [02:58:08] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 9.42, 9.70, 10.02 [02:58:22] CosmicAlpha, cool [02:59:02] dmehus: if this stays consistent I think I will consider https://phabricator.miraheze.org/T8608 resolved [02:59:03] [url] ⚓ T8608 MediaWiki performance issues | phabricator.miraheze.org [02:59:24] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.47, 7.91, 8.46 [03:00:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.35, 9.80, 9.91 [03:00:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.17, 9.47, 9.91 [03:02:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.30, 7.45, 8.39 [03:05:03] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.31, 7.60, 8.26 [03:08:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.04, 8.92, 9.32 [03:09:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6769d239f766...5017e0298b31 [03:09:44] [02miraheze/mw-config] 07Universal-Omega 035017e02 - set $wgJobRunRate to 0 [03:09:57] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to scsvg [03:10:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:10:05] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to scsvg - SUCCESS in 7s [03:10:06] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to skip [03:10:07] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to skip - SUCCESS in 0s [03:10:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:10:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:10:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:10:46] miraheze/mw-config - Universal-Omega the build passed. [03:11:21] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:11:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5017e0298b31...963c10ad5db4 [03:11:23] [02miraheze/mw-config] 07Universal-Omega 03963c10a - remove $wgJobRunRate from $wgConf [03:11:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.06, 8.72, 9.18 [03:11:56] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to scsvg [03:12:01] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to skip [03:12:03] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to skip - SUCCESS in 1s [03:12:08] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to scsvg - SUCCESS in 12s [03:12:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:12:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:12:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:12:26] miraheze/mw-config - Universal-Omega the build passed. [03:12:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:14:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.82, 9.32, 8.81 [03:14:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.57, 9.75, 9.56 [03:17:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.64, 9.61, 9.52 [03:19:38] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.48, 7.89, 8.46 [03:21:21] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:22:48] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.39, 9.95, 9.19 [03:22:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.18, 8.59, 8.88 [03:22:50] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.26, 9.66, 9.47 [03:26:21] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:26:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.71, 9.01, 8.75 [03:28:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.16, 9.22, 9.13 [03:29:23] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.06, 8.36, 8.58 [03:31:24] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.44, 9.52, 9.62 [03:31:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.40, 8.75, 9.13 [03:32:10] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.60, 8.09, 8.44 [03:34:07] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.17, 9.92, 9.70 [03:36:21] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:36:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.55, 8.75, 9.29 [03:39:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.34, 9.04, 8.83 [03:44:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.16, 8.61, 8.90 [03:45:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.05, 8.74, 8.80 [03:46:21] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:47:36] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.99, 8.19, 8.69 [03:50:21] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.57, 7.90, 8.49 [03:50:48] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.98, 7.65, 8.37 [03:56:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.34, 8.86, 8.76 [04:01:21] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:01:33] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.10, 7.46, 8.26 [04:02:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.80, 7.97, 8.41 [04:03:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.51, 3.50, 2.72 [04:06:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.69, 3.22, 2.75 [04:07:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.40, 8.33, 8.53 [04:10:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.78, 7.59, 8.20 [04:21:21] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:22:16] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.90, 8.59, 7.57 [04:22:16] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.24, 2.85 [04:23:36] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.68, 9.22, 8.54 [04:24:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.79, 3.46, 3.12 [04:24:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.82, 9.28, 8.45 [04:25:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.52, 3.59, 3.06 [04:25:05] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.28, 8.21, 7.61 [04:26:21] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:26:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.87, 2.90, 2.97 [04:27:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.84, 3.24, 3.01 [04:29:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.45, 8.50, 8.42 [04:35:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.80, 8.86, 8.60 [04:35:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.21, 9.28, 8.74 [04:36:21] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:37:59] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.32, 8.36, 8.43 [04:38:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.04, 8.98, 8.70 [04:41:18] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.04, 9.12, 8.78 [04:43:47] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.wikimicrofinanza.it All nameservers failed to answer the query. [04:43:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.14, 9.22, 8.87 [04:44:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.30, 9.03, 8.73 [04:44:27] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [04:46:21] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:49:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.94, 7.89, 8.34 [04:52:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.49, 8.90, 8.17 [04:52:06] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.82, 8.63, 8.58 [04:54:53] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.98, 8.40, 8.09 [04:54:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.29, 9.13, 8.81 [05:00:30] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.14, 7.79, 8.31 [05:11:36] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [05:13:50] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/us [05:13:50] thon3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query www.wikimicrofinanza.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:15:33] [02ssl] 07Universal-Omega closed pull request 03#469: Remove robloxapi.wiki - 13https://github.com/miraheze/ssl/pull/469 [05:22:26] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.95, 1.84, 1.36 [05:25:12] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.50, 1.30, 1.23 [05:26:21] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:31:21] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:00:51] CosmicAlpha, yeah definitely, SGTM :) [06:04:24] dmehus: sorry, what was that a reply to? [06:13:52] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [06:17:14] CosmicAlpha, where you said you might close that Phabricator task as resolved soon [06:17:19] if things remain constant [06:17:33] I'd maybe give it one full day of monitoring, then close [06:38:42] dmehus: oh right. Yep. I'll close tomorrow if it remains good still. [06:45:47] CosmicAlpha, okay, sounds good [06:46:01] * dmehus just realized Reception123 wasn't around all weekend [06:46:25] Was last night. At least talked with for a bit. [06:46:53] CosmicAlpha, yeah I guess briefly in my Saturday late morning (his evening/night time) [06:47:21] His morning also [06:47:22] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/u [06:47:22] ython3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wikimicrofinanza.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:47:55] dmehus: also as for Central auto login on custom domain wikis I think it should be a bit faster as a don't see a difference between now and pre-migration anymore. [06:59:32] CosmicAlpha, yeah, perhaps that will be resolved with the caching changes [06:59:49] CosmicAlpha, I'm going to try a global rename now [06:59:52] the real test... [07:01:22] Oh yeah... [07:02:01] It's a bit faster, but still not as super fast as it was last summer [07:02:21] dmehus: you mean central logins? [07:02:42] Or renames? [07:06:40] renames [07:07:00] The renames finished now, so definitely faster, but still could be a bit faster [07:07:06] but I can settle for this [07:08:54] dmehus: oh great! At least it's better. [07:10:06] Server load as been low for over 15 minutes now. That's the first time in months! [07:10:36] > OK - load average: 4.54, 5.34, 6.01 [07:10:48] I don't remember the last time I saw loads that low. [07:11:58] CosmicAlpha, yeah, definitely :) [07:12:02] I'd say at least 6 months [07:17:25] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [07:36:21] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:36:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.48, 3.13 [07:39:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 3.43, 3.14 [07:41:21] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:42:02] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.58, 2.92, 2.98 [07:51:21] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:52:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.49, 9.27, 8.00 [07:54:12] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.15, 10.77, 8.62 [07:55:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.71, 11.85, 8.72 [07:57:32] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.98, 8.51, 8.02 [08:00:21] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.16, 7.96, 7.89 [08:02:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.44, 8.87, 8.84 [08:05:27] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.72, 11.05, 9.68 [08:08:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.86, 9.34, 9.25 [08:11:21] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:16:21] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:18:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 4.00, 3.01, 2.43 [08:18:45] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.67, 7.90, 8.44 [08:21:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.09, 2.75, 2.41 [08:28:24] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.20, 7.58, 9.51 [08:31:21] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:36:36] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.34, 6.61, 8.25 [08:39:01] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.31, 7.66, 9.57 [08:44:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.87, 10.51, 10.14 [08:46:21] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:49:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.05, 9.65, 9.93 [08:51:21] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:57:59] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.64, 7.09, 8.46 [09:43:48] PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is CRITICAL: connect to address www.wikimicrofinanza.it and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [09:59:14] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/u [09:59:14] ython3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wikimicrofinanza.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:13:46] PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.wikimicrofinanza.it' expires in 14 day(s) (Mon 31 Jan 2022 22:54:41 GMT +0000). [10:19:55] [02mw-config] 07AgentIsai opened pull request 03#4395: Add sitenotice for closed wikis - 13https://github.com/miraheze/mw-config/pull/4395 [10:21:06] miraheze/mw-config - AgentIsai the build passed. [10:21:36] anyone around to merge? ^ [10:21:52] Is that still a problem? [10:22:02] JohnLewis: yes, it is [10:22:14] and we're still getting inquiries about it [10:22:17] Let me fix it... [10:22:50] Can you tell me a recent inquiry so I can look into a particular entry to understand why it is still a problem? [10:25:01] JohnLewis: https://meta.miraheze.org/wiki/Community_noticeboard#No_update_or_edit? - Most recent topic created asking about an erroneously closed wiki [10:25:03] [url] Community noticeboard - Miraheze Meta | meta.miraheze.org [10:25:15] but the user seems to have reopened the wiki themselves just now [10:25:32] Hm, okay [10:27:03] Morning everybody. [10:27:48] !log MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_closed = 0 AND wiki_closed_timestamp = NULL WHERE wiki_closed = 1; [10:27:50] Agent: fixed [10:27:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:28:02] JohnLewis: thank you! [10:28:03] [02mw-config] 07JohnFLewis commented on pull request 03#4395: Add sitenotice for closed wikis - 13https://github.com/miraheze/mw-config/pull/4395#issuecomment-1014366182 [10:28:07] [02mw-config] 07JohnFLewis closed pull request 03#4395: Add sitenotice for closed wikis - 13https://github.com/miraheze/mw-config/pull/4395 [10:29:21] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.wikimicrofinanza.it All nameservers failed to answer the query. [10:29:32] Agent: I'm resetting wiki caches now as well, so that change should start to be picked up on wikis shortlu [10:29:59] sounds good, thank you [10:41:59] !log ALTER TABLE searchindex ADD FULLTEXT INDEX si_title (si_title), ADD FULLTEXT INDEX si_text (si_text); [10:42:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:59:26] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [11:42:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.73, 2.92 [11:45:13] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.85, 3.38, 2.91 [11:46:21] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:56:21] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:59:11] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/772a46f71b77...c4a3db3543e1 [11:59:12] [02miraheze/puppet] 07Reception123 03c4a3db3 - remove www.mikrodev.com CSP whitelist entry (per T7886) [12:06:38] [02ssl] 07Reception123 closed pull request 03#470: Remove wiki.hausos.co - 13https://github.com/miraheze/ssl/pull/470 [12:06:39] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/f8875e5b39c5...8dc1fefe8665 [12:06:41] [02miraheze/ssl] 07Universal-Omega 038dc1fef - Remove wiki.hausos.co (#470) [12:06:48] [02ssl] 07Reception123 closed pull request 03#471: Remove wiki.3805.co.uk - 13https://github.com/miraheze/ssl/pull/471 [12:06:49] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/8dc1fefe8665...46af91038c04 [12:06:51] [02miraheze/ssl] 07Universal-Omega 0346af910 - Remove wiki.3805.co.uk (#471) [12:08:36] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/2b0f27deab37...bc44a6627bda [12:08:38] [02miraheze/CreateWiki] 07translatewiki 03bc44a66 - Localisation updates from https://translatewiki.net. [12:08:39] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/a5a55c23052f...b9bb66b04e63 [12:08:41] [02miraheze/ManageWiki] 07translatewiki 03b9bb66b - Localisation updates from https://translatewiki.net. [12:08:42] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+1/-0/±8] 13https://github.com/miraheze/MirahezeMagic/compare/55a7813f642b...903251f4850d [12:08:44] [02miraheze/MirahezeMagic] 07translatewiki 03903251f - Localisation updates from https://translatewiki.net. [12:11:05] !log RENAMED planita13wiki to planitawiki [12:11:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:13:00] miraheze/CreateWiki - translatewiki the build has errored. [12:18:26] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/8e40a6ca0b53...8ad1982d96a8 [12:18:27] [02miraheze/dns] 07Reception123 038ad1982 - add otogebase.com zone [12:19:38] miraheze/ManageWiki - translatewiki the build passed. [12:20:41] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±2] 13https://github.com/miraheze/ssl/compare/46af91038c04...faf094279cb6 [12:20:42] [02miraheze/ssl] 07Reception123 039837dbe - add wiki.rosestulipsandliberty.com cert [12:20:44] [02miraheze/ssl] 07Reception123 03faf0942 - add otogebase.com cert [12:22:55] miraheze/MirahezeMagic - translatewiki the build passed. [12:26:21] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:31:21] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:35:52] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.wikimicrofinanza.it All nameservers failed to answer the query. [13:00:49] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=dkptwiki /home/reception/dkwikipt --summary=Attribution: donkey-kong.fandom.com/pt/wiki (START) [13:00:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:00:53] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=dkptwiki /home/reception/dkwikipt --summary=Attribution: donkey-kong.fandom.com/pt/wiki (END - exit=0) [13:00:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:01:36] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=dkptwiki /home/reception/dkwikipt --summary=Attribution: donkey-kong.fandom.com/pt/wiki --search-recursively (START) [13:01:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:02:10] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=dkptwiki /home/reception/dkwikipt --summary=Attribution: donkey-kong.fandom.com/pt/wiki --search-recursively (END - exit=2) [13:02:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:02:16] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=dkptwiki /home/reception/dkwikipt --summary=Attribution:donkey-kong.fandom.com/pt/wiki --search-recursively (START) [13:02:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:05:43] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [13:13:08] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=dkptwiki /home/reception/dkwikipt --summary=Attribution:donkey-kong.fandom.com/pt/wiki --search-recursively (END - exit=0) [13:13:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:21:59] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.64, 3.02 [13:22:39] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.44, 9.63, 8.03 [13:23:57] [02mediawiki] 07RhinosF1 labeled pull request 03#4959: Bump extensions/RemovePII from `8c799ed` to `1b47a35` - 13https://github.com/miraheze/mediawiki/pull/4959 [13:24:21] [02mediawiki] 07RhinosF1 commented on pull request 03#4959: Bump extensions/RemovePII from `8c799ed` to `1b47a35` - 13https://github.com/miraheze/mediawiki/pull/4959#issuecomment-1014535148 [13:25:22] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.95, 1.68, 1.11 [13:25:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.76, 8.82, 8.00 [13:26:21] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:26:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.11, 5.55, 3.94 [13:27:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.62, 3.21 [13:28:23] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.07, 7.95, 7.80 [13:30:26] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.32, 3.05, 3.06 [13:30:52] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.45, 1.66, 1.28 [13:31:21] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:34:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.91, 3.67, 3.64 [13:39:06] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.79, 9.43, 8.38 [13:41:21] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:42:06] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.30, 8.61, 7.82 [13:43:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.73, 13.18, 9.92 [13:44:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.98, 7.62, 7.93 [13:44:48] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.56, 7.53, 7.52 [13:46:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.48, 2.86, 3.33 [13:49:20] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.49, 9.87, 9.42 [13:56:21] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:02:53] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.57, 7.52, 8.35 [14:10:04] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.06, 8.29, 7.76 [14:12:48] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.53, 8.25, 7.86 [14:31:21] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:32:24] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.22, 9.40, 8.51 [14:35:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.44, 10.13, 8.90 [14:35:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.02, 8.70, 8.33 [14:36:21] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:38:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.24, 9.76, 8.97 [14:41:21] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:43:14] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.28, 3.34, 2.57 [14:44:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.67, 8.29, 8.30 [14:44:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.97, 9.05, 8.06 [14:45:55] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.01, 2.69, 2.44 [14:47:25] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.91, 9.84, 8.52 [14:49:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.64, 9.38, 9.01 [14:50:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.75, 9.84, 8.73 [14:52:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.94, 9.00, 8.94 [14:53:15] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.36, 10.22, 9.03 [14:55:56] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.85, 9.27, 8.84 [14:58:23] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.27, 8.37, 8.43 [14:58:39] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=alicesoftwiki --search-recursively /home/reception/alicesoft --user Miraheze_maintenance_script --comment=Imported from alicesoft.fandom.com (START) [14:58:42] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=alicesoftwiki --search-recursively /home/reception/alicesoft --user Miraheze_maintenance_script --comment=Imported from alicesoft.fandom.com (END - exit=65280) [14:58:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:58:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:59:47] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=alicesoftwiki --search-recursively /home/reception/alicesoft --user Miraheze_maintenance_script --comment=Imported from alicesoft.fandom.com (START) [14:59:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:59:54] !log created local account for mh maint script user on alicesoftwiki [14:59:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:00:28] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.23, 8.08, 8.47 [15:04:39] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.13, 8.92, 8.61 [15:06:21] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:06:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.37, 9.28, 8.90 [15:07:59] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:08:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:09:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.27, 9.52, 9.12 [15:10:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.91, 8.78, 8.73 [15:13:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:15:28] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.66, 9.80, 9.33 [15:17:15] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.70, 8.79, 8.65 [15:17:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.78, 3.59, 3.09 [15:18:10] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.06, 9.31, 9.19 [15:18:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.94, 8.52, 8.15 [15:21:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.12, 7.42, 7.82 [15:22:55] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.88, 3.30, 3.16 [15:23:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:26:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.24, 10.07, 9.43 [15:27:14] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.30, 7.29, 8.16 [15:28:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.65, 10.12, 8.88 [15:28:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:29:12] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.52, 4.20, 3.40 [15:29:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.81, 8.95, 9.11 [15:29:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.08, 5.45, 4.28 [15:30:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.51, 4.78, 3.79 [15:31:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.13, 9.33, 8.79 [15:32:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.77, 3.71, 3.36 [15:33:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.34, 8.91, 8.58 [15:34:27] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.10, 10.42, 9.63 [15:34:59] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.56, 3.01, 3.14 [15:35:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.18, 3.55, 3.78 [15:35:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.46, 2.98, 3.34 [15:36:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.38, 8.94, 8.71 [15:38:16] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.46, 9.14, 9.03 [15:38:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.98, 4.04, 3.92 [15:38:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:39:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.80, 9.14, 8.73 [15:39:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.38, 8.51, 8.60 [15:41:10] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.42, 9.14, 9.04 [15:41:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.82, 3.88 [15:42:35] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.90, 8.17, 8.40 [15:42:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.08, 9.43, 8.93 [15:43:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:45:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.39, 8.76, 8.75 [15:45:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.37, 9.86, 9.92 [15:51:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.45, 10.69, 10.15 [15:52:16] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.38, 9.32, 8.98 [15:53:58] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.33, 7.98, 8.39 [15:55:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.06, 9.00, 8.91 [15:58:24] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.32, 3.14, 3.40 [15:58:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:03:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:06:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.19, 3.75, 3.42 [16:08:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:09:23] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.92, 9.34, 9.04 [16:12:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.29, 8.43, 8.76 [16:12:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.57, 9.13, 8.56 [16:12:47] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.81, 3.85, 3.62 [16:15:04] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.33, 7.72, 8.42 [16:15:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.66, 8.29, 8.32 [16:17:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.18, 8.71, 9.78 [16:18:13] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.82, 3.15, 3.36 [16:18:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:18:49] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.wikimicrofinanza.it All nameservers failed to answer the query. [16:25:26] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.49, 9.42, 9.65 [16:25:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.52, 3.64, 3.29 [16:27:39] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.78, 3.64, 3.44 [16:28:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.19, 4.33, 3.73 [16:28:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.04, 3.51, 3.31 [16:28:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:29:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.47, 8.79, 8.46 [16:29:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.66, 9.19, 8.80 [16:31:12] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.63, 8.60, 9.20 [16:31:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.08, 3.73, 3.39 [16:32:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.89, 9.50, 8.77 [16:32:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.73, 10.25, 9.26 [16:33:32] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.30, 4.12, 3.70 [16:33:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:34:24] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.56, 3.30, 3.29 [16:34:30] Reception123: alot of ssl certificates expire in 1 day. We really need to get the ssl renewals working again. I don't think it works because of IPV6 when github is IPv4. [16:35:05] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.87, 9.49, 8.95 [16:35:12] *few days. [16:35:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.42, 8.74, 8.83 [16:36:21] Also can https://github.com/miraheze/ssl/pull/473 and https://github.com/miraheze/ssl/pull/475 + their dns counterparts be merged? [16:36:22] [url] remove rangpurpedia.xyz cert (expired domain) by Reception123 · Pull Request #473 · miraheze/ssl · GitHub | github.com [16:36:23] [url] Remove sportsanalytics.wiki by Universal-Omega · Pull Request #475 · miraheze/ssl · GitHub | github.com [16:36:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.70, 3.75, 3.62 [16:37:04] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.18, 10.44, 9.76 [16:39:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.73, 4.58, 3.93 [16:41:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.85, 10.14, 9.28 [16:42:24] PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is CRITICAL: connect to address www.wikimicrofinanza.it and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:43:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.54, 10.51, 9.60 [16:43:59] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.24, 9.64, 8.52 [16:44:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.57, 4.26, 3.69 [16:44:56] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.08, 8.73, 8.22 [16:45:46] !log set cloud3,4,5 to terminate at end of their contracts [16:45:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:46:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.97, 9.43, 9.36 [16:47:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.01, 8.87, 8.42 [16:47:25] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.49, 9.36, 8.71 [16:47:50] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.74, 8.29, 8.18 [16:49:06] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [16:49:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.75, 8.36, 8.30 [16:50:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.61, 3.59, 3.88 [16:51:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.59, 3.20, 3.81 [16:52:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.30, 3.29, 3.53 [16:54:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.74, 9.58, 10.00 [16:55:25] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.43, 2.89, 3.33 [16:55:41] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.47, 8.42, 8.48 [16:56:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.11, 3.19, 3.55 [16:57:05] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.38, 10.02, 10.11 [16:58:28] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.51, 9.73, 9.86 [16:59:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.61, 3.25, 3.51 [16:59:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.91, 9.70, 9.99 [17:02:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.45, 8.47, 8.52 [17:03:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.60, 3.62 [17:03:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:03:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.79, 8.81, 8.81 [17:04:15] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.48, 9.93, 9.84 [17:05:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.96, 9.25, 8.44 [17:05:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.96, 8.12, 8.39 [17:05:49] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.62, 10.68, 10.28 [17:07:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.72, 9.11, 9.54 [17:08:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.93, 3.61, 3.41 [17:08:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.77, 4.17, 3.78 [17:08:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.58, 9.71, 8.79 [17:08:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:09:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.98, 9.16, 9.02 [17:11:07] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 3.29, 3.33 [17:11:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.95, 3.98, 3.76 [17:11:13] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.31, 9.97, 9.02 [17:12:17] PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.wikimicrofinanza.it' expires in 14 day(s) (Mon 31 Jan 2022 22:54:41 GMT +0000). [17:14:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.50, 9.91, 9.17 [17:14:50] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.46, 3.38, 3.72 [17:16:45] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.31, 2.76, 3.27 [17:18:40] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.16, 8.67, 8.85 [17:20:24] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.36, 3.40, 3.58 [17:21:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.04, 8.74, 8.86 [17:23:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:24:02] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.53, 9.40, 9.05 [17:26:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.02, 8.55, 8.77 [17:27:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.40, 9.60, 9.18 [17:28:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:29:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.42, 2.90, 3.39 [17:30:06] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.63, 9.03, 9.03 [17:32:13] CosmicAlpha: how can we fix them then? [17:32:42] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.94, 9.57, 9.07 [17:33:27] Reception123: it needs to use something other than ssh [17:34:04] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.47, 7.60, 8.33 [17:36:58] Reception123: https://techglimpse.com/git-push-github-token-based-passwordless/ [17:36:58] [url] git push using GitHub token [Deprecating password authentication] - Techglimpse | techglimpse.com [17:38:01] Or set the proxy https://stackoverflow.com/questions/19161960/connect-with-ssh-through-a-proxy [17:38:02] [url] Connect with SSH through a proxy - Stack Overflow | stackoverflow.com [17:38:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.71, 9.13, 9.08 [17:40:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.26, 9.51, 8.98 [17:41:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.17, 9.75, 9.30 [17:42:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 3.56, 3.18 [17:44:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.35, 9.59, 9.28 [17:46:35] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.77, 9.50, 8.66 [17:47:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.67, 4.15, 3.67 [17:47:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.78, 9.71, 9.38 [17:47:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.38, 9.82, 9.53 [17:48:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.36, 3.38, 3.30 [17:49:24] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.34, 8.98, 8.60 [17:49:31] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.82, 9.26, 8.37 [17:49:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.39, 9.03, 9.07 [17:50:43] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.52, 10.38, 9.78 [17:53:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.28, 9.74, 9.61 [17:55:02] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.26, 7.88, 8.29 [17:55:09] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.67, 7.93, 8.06 [17:58:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.21, 7.53, 8.27 [18:02:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.13, 3.74, 3.85 [18:08:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 3.52, 3.67 [18:08:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:10:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.50, 3.54, 3.66 [18:16:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.52, 3.79, 3.65 [18:18:25] RhinosF1: ah I see [18:18:36] we'll need to figure that out then [18:18:50] Reception123: it's a fairly simple change [18:20:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.89, 3.39, 3.23 [18:22:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.98, 3.55, 3.59 [18:24:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.28, 2.20, 1.68 [18:27:05] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.59, 1.93, 1.66 [18:28:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 3.75, 3.63 [18:28:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:29:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.92, 3.86, 3.42 [18:31:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.29, 3.05, 3.38 [18:31:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.04, 3.06, 3.19 [18:31:54] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.51, 9.06, 8.22 [18:33:08] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.49, 1.63, 1.61 [18:33:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:34:15] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.84, 10.91, 9.88 [18:34:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.25, 9.25, 8.45 [18:37:41] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.83, 8.33, 8.21 [18:38:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:41:55] PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is CRITICAL: connect to address www.wikimicrofinanza.it and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:45:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.99, 9.29, 9.79 [18:47:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.77, 3.45, 3.47 [18:48:18] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.84, 8.07, 8.39 [18:49:58] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.33, 3.16, 3.34 [18:50:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.75, 8.95, 9.78 [18:53:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:55:25] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.80, 1.89, 1.74 [18:55:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.59, 8.94, 8.36 [18:55:53] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.04, 9.33, 9.69 [18:58:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:01:23] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.26, 1.19, 1.53 [19:01:53] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.65, 8.35, 8.28 [19:02:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.98, 3.56, 3.39 [19:02:36] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.34, 10.35, 9.82 [19:02:50] JohnLewis, paladox: logsal is dead [19:03:09] or not [19:03:20] no moveBatch just isn't lng [19:03:34] !log moving Requests for adoption -> reopening [19:03:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:04:13] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/f4f9db49242f...b57ada3bd9e8 [19:04:15] [02miraheze/mediawiki] 07Universal-Omega 03b57ada3 - Update DiscordNotifications [19:05:24] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.91, 3.11, 3.23 [19:05:28] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 5.84, 8.46, 9.20 [19:05:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/680ee97b3380 [19:05:56] [02miraheze/mw-config] 07Universal-Omega 03680ee97 - set $wgDiscordCurlProxy [19:05:57] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [19:05:59] [02mw-config] 07Universal-Omega opened pull request 03#4396: set $wgDiscordCurlProxy - 13https://github.com/miraheze/mw-config/pull/4396 [19:06:37] !log [universalomega@test101] starting deploy of {'world': True} to skip [19:06:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:02] miraheze/mw-config - Universal-Omega the build passed. [19:08:18] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.89, 9.22, 9.34 [19:10:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.67, 9.25, 9.94 [19:10:40] !log [universalomega@test101] finished deploy of {'world': True} to skip - SUCCESS in 242s [19:11:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:12:08] PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.wikimicrofinanza.it' expires in 14 day(s) (Mon 31 Jan 2022 22:54:41 GMT +0000). [19:12:38] !log [universalomega@test101] starting deploy of {'config': True} to skip [19:12:39] !log [universalomega@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [19:12:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.27, 9.37, 8.87 [19:13:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:13:27] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=alicesoftwiki --search-recursively /home/reception/alicesoft --user Miraheze_maintenance_script --comment=Imported from alicesoft.fandom.com (END - exit=0) [19:13:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:14:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:14:08] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.81, 8.95, 8.04 [19:15:15] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.49, 12.85, 10.76 [19:16:59] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.76, 8.24, 7.91 [19:18:33] PROBLEM - test101 Puppet on test101 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 30 minutes ago with 0 failures [19:20:17] [02mw-config] 07Universal-Omega closed pull request 03#4396: set $wgDiscordCurlProxy - 13https://github.com/miraheze/mw-config/pull/4396 [19:20:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/963c10ad5db4...738f38a7b1da [19:20:20] [02miraheze/mw-config] 07Universal-Omega 03738f38a - set $wgDiscordCurlProxy (#4396) [19:20:21] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [19:20:23] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [19:20:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.06, 9.26, 9.86 [19:21:19] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:21:23] miraheze/mw-config - Universal-Omega the build passed. [19:22:26] !log [@mwtask111] starting deploy of {'config': True} to scsvg [19:22:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.14, 9.38, 9.49 [19:22:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:22:38] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 12s [19:23:02] !log [universalomega@mwtask111] starting deploy of {'world': True, 'proxy': True} to scsvg [19:23:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:23:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:23:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.55, 9.68, 9.90 [19:26:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.46, 3.40, 3.04 [19:27:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.04, 9.84, 9.28 [19:28:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.85, 8.67, 9.54 [19:29:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.61, 3.13, 3.01 [19:30:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.47, 8.94, 9.01 [19:30:12] !log [universalomega@mwtask111] finished deploy of {'world': True, 'proxy': True} to scsvg - SUCCESS in 428s [19:30:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:32:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.72, 3.48, 3.17 [19:35:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.82, 2.94, 3.03 [19:35:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.19, 10.60, 9.66 [19:38:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.63, 9.19, 9.70 [19:39:27] PROBLEM - test101 Puppet on test101 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 51 minutes ago with 0 failures [19:50:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.84, 9.97, 9.96 [19:51:37] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to skip [19:51:39] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to skip - SUCCESS in 1s [19:51:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:51:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:51:58] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to skip [19:51:59] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True, 'ignoretime': True} to skip - SUCCESS in 0s [19:52:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:08] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.31, 8.68, 8.20 [19:53:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.41, 8.84, 8.56 [19:53:40] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:54:28] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.81, 10.82, 9.99 [19:55:35] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.50, 9.54, 9.68 [19:56:09] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.32, 10.55, 9.27 [19:58:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.43, 9.28, 9.54 [19:58:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.91, 9.50, 8.73 [19:58:57] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.00, 9.77, 9.18 [20:01:13] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.52, 7.98, 8.48 [20:01:45] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.60, 10.39, 9.52 [20:03:53] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.wikimicrofinanza.it All nameservers failed to answer the query. [20:04:32] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.27, 9.20, 9.19 [20:08:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.32, 9.42, 9.96 [20:08:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.17, 9.15, 9.68 [20:09:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.81, 9.64, 9.47 [20:12:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.35, 8.06, 8.41 [20:15:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.81, 9.57, 9.51 [20:16:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.66, 9.65, 9.74 [20:18:15] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.51, 10.47, 9.85 [20:19:39] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.83, 9.11, 9.54 [20:19:55] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.20, 9.70, 9.54 [20:21:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.56, 9.83, 9.73 [20:25:13] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.96, 9.66, 9.65 [20:25:28] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.01, 9.42, 9.59 [20:25:34] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.58, 8.76, 8.67 [20:26:40] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.58, 9.36, 9.49 [20:28:27] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.14, 9.71, 9.66 [20:31:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.42, 9.33, 9.51 [20:32:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.63, 8.88, 9.34 [20:32:56] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.78, 11.19, 9.79 [20:34:26] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/us [20:34:26] thon3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query www.wikimicrofinanza.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:34:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.64, 9.06, 8.69 [20:35:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.64, 10.87, 10.08 [20:36:51] [02puppet] 07Universal-Omega opened pull request 03#2351: ssl-acme: proxy through bastion - 13https://github.com/miraheze/puppet/pull/2351 [20:37:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.37, 8.91, 8.69 [20:38:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.12, 9.27, 9.60 [20:38:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.77, 9.47, 9.60 [20:38:53] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.95, 3.80, 3.35 [20:40:13] Hey guys - really mad right now. I literally fixed the DNS misconfiguration yesterday and you still disabled my subdomain!?!?!?! https://robloxapi.wiki/ [20:41:06] At least it looks like it's disabled - maybe something else happened? [20:41:20] it should not be disabled. Looking [20:41:41] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.45, 3.40, 3.27 [20:42:45] it's expired [20:43:14] how? I made sure that cloudflare was not proxying the CNAME [20:43:21] What do I need to do to fix this? [20:45:34] I don't think you'll have to do anything, it should just need to be renewed on our end [20:45:37] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/faf094279cb6...fa1faa05c9aa [20:45:39] [02miraheze/ssl] 07paladox 03fa1faa0 - Update robloxapi.wiki.crt [20:46:10] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.04, 9.54, 8.98 [20:48:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.57, 4.12, 3.63 [20:48:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.46, 9.87, 9.25 [20:48:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.55, 3.27 [20:49:12] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.90, 9.48, 9.05 [20:49:45] I think you fixed it! Thanks so much! [20:51:08] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.05, 3.80, 3.60 [20:51:26] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.38, 9.56, 9.24 [20:51:42] We did [20:51:51] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.15, 2.95, 3.09 [20:51:58] thanks paladox! [20:52:02] Voidwalker: it needs making to work with the proxy [20:52:06] It doesn't push anymore [20:52:16] It could be fixed easily [20:53:10] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.10, 7.58, 8.40 [20:54:05] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.13, 3.12, 3.37 [20:58:06] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.27, 7.54, 8.34 [20:59:23] https://github.com/miraheze/puppet/pull/2351/files I'm assuming? [20:59:24] [url] ssl-acme: proxy through bastion by Universal-Omega · Pull Request #2351 · miraheze/puppet · GitHub | github.com [20:59:48] [02puppet] 07Universal-Omega synchronize pull request 03#2351: ssl-acme: proxy through bastion - 13https://github.com/miraheze/puppet/pull/2351 [21:00:09] Voidwalker: yes [21:00:14] Exactly that [21:04:05] RECOVERY - robloxapi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'robloxapi.wiki' will expire on Sun 17 Apr 2022 19:44:52 GMT +0000. [21:04:22] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.64, 10.39, 9.68 [21:04:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.13, 9.11, 8.80 [21:05:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.26, 7.53, 8.34 [21:07:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.42, 9.29, 9.41 [21:09:13] [02puppet] 07Universal-Omega synchronize pull request 03#2351: ssl-acme: proxy through bastion - 13https://github.com/miraheze/puppet/pull/2351 [21:09:14] [02puppet] 07Universal-Omega synchronize pull request 03#2351: ssl-acme: proxy through bastion - 13https://github.com/miraheze/puppet/pull/2351 [21:10:20] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.02, 7.96, 8.43 [21:11:08] [02puppet] 07Universal-Omega synchronize pull request 03#2351: ssl-acme: proxy through bastion - 13https://github.com/miraheze/puppet/pull/2351 [21:11:34] RhinosF1: it does not work though. port 22 is not allowed in bastion it seems. [21:13:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:16:45] we will probably need to do with an access token [21:19:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.07, 8.64, 9.72 [21:21:08] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.81, 9.29, 9.14 [21:22:30] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.14, 8.91, 9.61 [21:24:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.11, 8.95, 9.05 [21:24:03] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.36, 9.09, 8.51 [21:24:28] !log sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php [21:24:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:26:34] CosmicAlpha: ok [21:26:37] It should [21:26:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.96, 9.64, 8.83 [21:27:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.70, 3.76, 3.26 [21:28:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:29:56] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.17, 3.23, 3.12 [21:37:32] [02puppet] 07Universal-Omega closed pull request 03#2351: ssl-acme: proxy through bastion - 13https://github.com/miraheze/puppet/pull/2351 [21:41:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.03, 9.81, 9.37 [21:42:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.27, 9.44, 9.88 [21:42:42] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/89bfcd31490b [21:42:44] [02miraheze/mw-config] 07Universal-Omega 0389bfcd3 - set `$wgRSSProxy` [21:42:45] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [21:42:47] [02mw-config] 07Universal-Omega opened pull request 03#4397: set `$wgRSSProxy` - 13https://github.com/miraheze/mw-config/pull/4397 [21:43:41] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:43:45] miraheze/mw-config - Universal-Omega the build passed. [21:44:11] [02mw-config] 07Universal-Omega closed pull request 03#4397: set `$wgRSSProxy` - 13https://github.com/miraheze/mw-config/pull/4397 [21:44:13] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/738f38a7b1da...1f1b91ba0f1c [21:44:14] [02miraheze/mw-config] 07Universal-Omega 031f1b91b - set `$wgRSSProxy` (#4397) [21:44:16] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [21:44:17] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [21:44:26] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to scsvg [21:44:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:44:36] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to scsvg - SUCCESS in 10s [21:44:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:44:56] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to skip [21:44:57] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to skip - SUCCESS in 1s [21:45:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:45:15] miraheze/mw-config - Universal-Omega the build passed. [21:45:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:46:47] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.03, 9.42, 8.98 [21:48:28] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.99, 10.31, 10.02 [21:48:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:49:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.82, 9.80, 9.21 [21:49:43] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.60, 9.46, 9.45 [21:50:51] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.29, 8.68, 8.34 [21:51:42] [02puppet] 07Universal-Omega opened pull request 03#2352: Revert "absent manageInactiveWikis cron" - 13https://github.com/miraheze/puppet/pull/2352 [21:52:35] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.17, 10.13, 9.43 [21:53:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.35, 8.73, 8.25 [21:55:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.78, 9.50, 9.40 [21:55:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.75, 3.39 [21:56:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.87, 3.61, 3.25 [21:56:41] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.77, 8.24, 8.15 [21:56:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.18, 4.05, 3.41 [21:58:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.34, 4.24, 3.63 [21:59:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/98544e554867 [21:59:15] [02miraheze/mw-config] 07Universal-Omega 0398544e5 - move `$wgVirtualRestConfig` to defines.php [21:59:17] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [21:59:18] [02mw-config] 07Universal-Omega opened pull request 03#4398: move `$wgVirtualRestConfig` to defines.php - 13https://github.com/miraheze/mw-config/pull/4398 [21:59:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/98544e554867...c3c2edcb4808 [21:59:47] [02miraheze/mw-config] 07Universal-Omega 03c3c2edc - Update LocalSettings.php [21:59:49] [02mw-config] 07Universal-Omega synchronize pull request 03#4398: move `$wgVirtualRestConfig` to defines.php - 13https://github.com/miraheze/mw-config/pull/4398 [21:59:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.04, 3.61, 3.36 [21:59:57] [02mw-config] 07Universal-Omega edited pull request 03#4398: move `$wgVirtualRestConfig` to Defines.php - 13https://github.com/miraheze/mw-config/pull/4398 [22:00:07] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.67, 1.94, 1.53 [22:00:21] miraheze/mw-config - Universal-Omega the build passed. [22:00:48] miraheze/mw-config - Universal-Omega the build passed. [22:01:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.67, 9.45, 9.46 [22:01:40] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.71, 3.53, 3.46 [22:02:41] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.81, 3.40, 3.33 [22:02:47] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.03, 1.52, 1.43 [22:05:11] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.46, 8.17, 8.49 [22:05:20] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [22:05:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.98, 9.31, 10.00 [22:06:53] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.25, 9.46, 9.31 [22:06:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.63, 9.68, 9.66 [22:07:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.39, 4.51, 3.78 [22:07:22] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.55, 3.74, 3.44 [22:09:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.65, 9.15, 9.23 [22:11:29] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.23, 9.63, 9.92 [22:12:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.32, 3.53, 3.43 [22:12:41] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.26, 9.68, 9.42 [22:13:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.89, 9.89, 8.63 [22:14:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.07, 9.63, 9.88 [22:15:02] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 3.30, 3.37 [22:15:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.48, 9.38, 9.37 [22:15:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.95, 3.77, 3.66 [22:18:09] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.68, 9.94, 9.55 [22:18:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.81, 3.68, 4.00 [22:20:57] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.67, 9.65, 9.49 [22:21:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.87, 3.08, 3.35 [22:22:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1f1b91ba0f1c...255b1c64831f [22:22:19] [02miraheze/mw-config] 07Universal-Omega 03255b1c6 - Add StructuredNavigation to extension-list [22:22:43] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.25, 10.06, 9.95 [22:23:19] miraheze/mw-config - Universal-Omega the build passed. [22:25:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.71, 8.78, 8.31 [22:25:32] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.72, 9.50, 9.74 [22:25:50] [02mw-config] 07Universal-Omega closed pull request 03#4398: move `$wgVirtualRestConfig` to Defines.php - 13https://github.com/miraheze/mw-config/pull/4398 [22:25:51] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/255b1c64831f...1e3094fa4c16 [22:25:53] [02miraheze/mw-config] 07Universal-Omega 031e3094f - move `$wgVirtualRestConfig` to Defines.php (#4398) [22:25:54] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [22:25:56] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [22:26:51] miraheze/mw-config - Universal-Omega the build passed. [22:27:14] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.98, 9.22, 9.43 [22:27:52] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.62, 7.91, 8.06 [22:28:23] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 9.95, 10.17, 9.97 [22:29:49] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.35, 2.68, 3.32 [22:31:07] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 5.92, 8.47, 9.36 [22:32:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.26, 7.27, 8.41 [22:33:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:33:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/915ff10382b8 [22:33:54] [02miraheze/mw-config] 07Universal-Omega 03915ff10 - Set `$wgCopyUploadProxy` [22:33:55] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [22:33:57] [02mw-config] 07Universal-Omega opened pull request 03#4399: Set `$wgCopyUploadProxy` - 13https://github.com/miraheze/mw-config/pull/4399 [22:34:25] [02mw-config] 07Universal-Omega closed pull request 03#4399: Set `$wgCopyUploadProxy` - 13https://github.com/miraheze/mw-config/pull/4399 [22:34:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1e3094fa4c16...8e3792f8fc7c [22:34:28] [02miraheze/mw-config] 07Universal-Omega 038e3792f - Set `$wgCopyUploadProxy` (#4399) [22:34:29] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [22:34:31] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [22:34:59] miraheze/mw-config - Universal-Omega the build passed. [22:35:30] miraheze/mw-config - Universal-Omega the build passed. [22:35:38] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.25, 8.38, 8.77 [22:38:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.26, 8.28, 8.67 [22:38:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/5d3dd0954297 [22:38:58] [02miraheze/mw-config] 07Universal-Omega 035d3dd09 - Set `wgHTTPProxy` [22:39:00] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [22:39:02] [02mw-config] 07Universal-Omega opened pull request 03#4400: Set `$wgHTTPProxy` - 13https://github.com/miraheze/mw-config/pull/4400 [22:39:16] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.wikimicrofinanza.it All nameservers failed to answer the query. [22:39:27] [02mw-config] 07Universal-Omega closed pull request 03#4400: Set `$wgHTTPProxy` - 13https://github.com/miraheze/mw-config/pull/4400 [22:39:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8e3792f8fc7c...7d71844b0132 [22:39:30] [02miraheze/mw-config] 07Universal-Omega 037d71844 - Set `$wgHTTPProxy` (#4400) [22:39:32] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [22:39:33] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [22:40:11] miraheze/mw-config - Universal-Omega the build passed. [22:40:34] miraheze/mw-config - Universal-Omega the build passed. [22:40:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.15, 7.97, 8.48 [22:41:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.53, 8.97, 8.85 [22:42:08] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.41, 7.52, 8.34 [22:44:04] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.83, 7.55, 8.32 [22:48:16] !log [@test101] starting deploy of {'config': True} to skip [22:48:17] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:48:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:48:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:52:12] !log [@mwtask111] starting deploy of {'config': True} to scsvg [22:52:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:52:20] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 8s [22:52:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:53:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:55:42] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.08, 8.31, 8.08 [22:58:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.83, 7.84, 7.95 [22:58:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:59:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.80, 8.68, 8.44 [23:00:01] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:03] !log [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg [23:00:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:15] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.09, 9.27, 8.70 [23:03:35] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.14, 3.76, 3.00 [23:05:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.57, 8.46, 8.47 [23:06:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.51, 3.17, 2.88 [23:08:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:09:04] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.59, 2.59, 2.71 [23:09:31] PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/us [23:09:31] thon3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query www.wikimicrofinanza.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:10:10] [02mw-config] 07Universal-Omega closed pull request 03#4376: Add $wgWBQualityConstraints* to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4376 [23:10:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/7d71844b0132...e4ad4edba455 [23:10:13] [02miraheze/mw-config] 07lens0021 03e4ad4ed - Add $wgWBQualityConstraints* to ManageWikiSettings (#4376) [23:11:16] miraheze/mw-config - Universal-Omega the build passed. [23:18:15] !log [@test101] starting deploy of {'config': True} to skip [23:18:16] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [23:18:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:18:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:18:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:23:13] !log [@mwtask111] starting deploy of {'config': True} to scsvg [23:23:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:23:23] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 9s [23:23:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:24:00] !log [@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 1439s [23:24:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:25:18] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.77, 8.36, 8.16 [23:26:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.38, 11.22, 9.29 [23:26:48] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 4.13, 3.51 [23:27:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/5ebed08572d8 [23:27:36] [02miraheze/mw-config] 07Universal-Omega 035ebed08 - Add StructuredNavigation to ManageWikiExtensions [23:27:38] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [23:27:44] !log [@mwtask111] finished deploy of {'l10nupdate': True} to scsvg - SUCCESS in 1661s [23:27:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:17] [02mw-config] 07Universal-Omega opened pull request 03#4401: Add StructuredNavigation to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4401 [23:28:40] alerting : [FIRING:2] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:29:21] miraheze/mw-config - Universal-Omega the build passed. [23:29:36] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.76, 3.21, 3.26 [23:30:57] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.51, 7.81, 8.09 [23:31:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5ebed08572d8...2a8d92581e02 [23:31:44] [02miraheze/mw-config] 07Universal-Omega 032a8d925 - Update ManageWikiExtensions.php [23:31:46] [02mw-config] 07Universal-Omega synchronize pull request 03#4401: Add StructuredNavigation to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4401 [23:32:47] miraheze/mw-config - Universal-Omega the build passed. [23:36:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2a8d92581e02...44569ea9017d [23:36:02] [02miraheze/mw-config] 07Universal-Omega 0344569ea - Update ManageWikiExtensions.php [23:36:03] [02mw-config] 07Universal-Omega synchronize pull request 03#4401: Add StructuredNavigation to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4401 [23:36:31] PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:37:03] miraheze/mw-config - Universal-Omega the build passed. [23:37:33] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:37:53] PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:39:38] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.929 second response time [23:40:01] RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [23:43:41] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 6.109 second response time [23:44:01] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 3.594 second response time [23:45:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/7930c44aa74d [23:45:11] [02miraheze/mw-config] 07Universal-Omega 037930c44 - Fix StructuredDiscussions' namespaces [23:45:13] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-6 - 13https://github.com/miraheze/mw-config [23:45:14] [02mw-config] 07Universal-Omega opened pull request 03#4402: Fix StructuredDiscussions' namespaces - 13https://github.com/miraheze/mw-config/pull/4402 [23:46:16] miraheze/mw-config - Universal-Omega the build passed. [23:49:40] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 0.38, 2.23, 1.85 [23:52:27] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.41, 1.39, 1.59 [23:53:40] alerting : [FIRING:1] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:57:13] [02puppet] 07Universal-Omega synchronize pull request 03#2305: MediaWiki: Use Debian composer package - 13https://github.com/miraheze/puppet/pull/2305 [23:57:28] [02puppet] 07Universal-Omega synchronize pull request 03#2305: MediaWiki: Use Debian composer package - 13https://github.com/miraheze/puppet/pull/2305 [23:57:51] [02puppet] 07Universal-Omega synchronize pull request 03#2305: MediaWiki: Use Debian composer package - 13https://github.com/miraheze/puppet/pull/2305