[00:02:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:02:35] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 23 failures. Last run 2 minutes ago with 23 failures. Failed resources (up to 3 shown): File[/etc/modprobe.d/nf_conntrack.conf],File[/usr/lib/nagios/plugins/check_conntrack],File[/usr/lib/nagios/plugins/check_ferm],File[/usr/lib/nagios/plugins/check_smart] [00:04:27] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.56, 2.58, 1.72 [00:06:27] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.81, 2.55, 1.81 [00:07:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:11:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.45, 3.25 [00:13:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.83, 23.12, 24.00 [00:15:18] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 38.33, 29.50, 26.25 [00:23:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.17, 3.34, 3.31 [00:29:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.85, 3.54, 3.37 [00:30:34] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [00:33:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.96, 3.33, 3.31 [00:37:51] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.61, 7.29, 6.16 [00:42:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:43:51] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.60, 5.74, 5.91 [00:47:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:48:58] PROBLEM - bacula2 Bacula Private Git on bacula2 is WARNING: WARNING: Full, 6775 files, 57.16MB, 2021-12-12 00:46:00 (1.1 weeks ago) [00:53:24] [02mediawiki] 07dependabot[bot] opened pull request 03#4679: Bump extensions/MatomoAnalytics from `4564cb1` to `0424557` - 13https://git.io/JDHS7 [00:53:25] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MatomoAnalytics-0424557 [+0/-0/±1] 13https://git.io/JDHS5 [00:53:27] [02miraheze/mediawiki] 07dependabot[bot] 035f9e4f4 - Bump extensions/MatomoAnalytics from `4564cb1` to `0424557` [00:53:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MatomoAnalytics-0424557 - 13https://git.io/vbL5b [00:53:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/RemovePII-8c799ed [+0/-0/±1] 13https://git.io/JDHSA [00:53:31] [02miraheze/mediawiki] 07dependabot[bot] 0337fa6ab - Bump extensions/RemovePII from `41a11a3` to `8c799ed` [00:53:33] [02mediawiki] 07dependabot[bot] opened pull request 03#4680: Bump extensions/RemovePII from `41a11a3` to `8c799ed` - 13https://git.io/JDHSx [00:53:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/RemovePII-8c799ed - 13https://git.io/vbL5b [00:53:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/BlogPage-a1b055b [+0/-0/±1] 13https://git.io/JDHSh [00:53:37] [02miraheze/mediawiki] 07dependabot[bot] 03b27af3c - Bump extensions/BlogPage from `f7526b8` to `a1b055b` [00:53:39] [02mediawiki] 07dependabot[bot] opened pull request 03#4681: Bump extensions/BlogPage from `f7526b8` to `a1b055b` - 13https://git.io/JDHSj [00:53:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/BlogPage-a1b055b - 13https://git.io/vbL5b [00:53:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SocialProfile-952f5d4 [+0/-0/±1] 13https://git.io/JDH9f [00:53:43] [02miraheze/mediawiki] 07dependabot[bot] 030f52cdf - Bump extensions/SocialProfile from `da33f37` to `952f5d4` [00:53:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SocialProfile-952f5d4 - 13https://git.io/vbL5b [00:53:46] [02mediawiki] 07dependabot[bot] opened pull request 03#4682: Bump extensions/SocialProfile from `da33f37` to `952f5d4` - 13https://git.io/JDH9J [00:53:48] [02mediawiki] 07dependabot[bot] commented on pull request 03#4656: Bump extensions/SocialProfile from `da33f37` to `0b143d8` - 13https://git.io/JDH9U [00:53:49] [02mediawiki] 07dependabot[bot] closed pull request 03#4656: Bump extensions/SocialProfile from `da33f37` to `0b143d8` - 13https://git.io/JD259 [00:53:51] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/SocialProfile-0b143d8 - 13https://git.io/vbL5b [00:53:52] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/SocialProfile-0b143d8 [00:53:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/FancyBoxThumbs-256e89c [+0/-0/±1] 13https://git.io/JDH9k [00:53:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.65, 3.37 [00:53:55] [02miraheze/mediawiki] 07dependabot[bot] 03dd5696f - Bump extensions/FancyBoxThumbs from `ac84351` to `256e89c` [00:53:57] [02mediawiki] 07dependabot[bot] opened pull request 03#4683: Bump extensions/FancyBoxThumbs from `ac84351` to `256e89c` - 13https://git.io/JDH9I [00:53:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/FancyBoxThumbs-256e89c - 13https://git.io/vbL5b [00:54:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Translate-e189bf4 [+0/-0/±1] 13https://git.io/JDH9Y [00:54:01] [02miraheze/mediawiki] 07dependabot[bot] 03bba76df - Bump extensions/Translate from `0babb26` to `e189bf4` [00:54:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Translate-e189bf4 - 13https://git.io/vbL5b [00:54:04] [02mediawiki] 07dependabot[bot] opened pull request 03#4684: Bump extensions/Translate from `0babb26` to `e189bf4` - 13https://git.io/JDH9O [00:54:06] [02mediawiki] 07dependabot[bot] commented on pull request 03#4628: Bump extensions/Translate from `0babb26` to `6a17c07` - 13https://git.io/JDH93 [00:54:07] [02mediawiki] 07dependabot[bot] closed pull request 03#4628: Bump extensions/Translate from `0babb26` to `6a17c07` - 13https://git.io/JD27H [00:54:09] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Translate-6a17c07 - 13https://git.io/vbL5b [00:54:10] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Translate-6a17c07 [00:54:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Monaco-a07ea87 [+0/-0/±1] 13https://git.io/JDH9s [00:54:13] [02miraheze/mediawiki] 07dependabot[bot] 03e15aa93 - Bump skins/Monaco from `90ccaf8` to `a07ea87` [00:54:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Monaco-a07ea87 - 13https://git.io/vbL5b [00:54:16] [02mediawiki] 07dependabot[bot] opened pull request 03#4685: Bump skins/Monaco from `90ccaf8` to `a07ea87` - 13https://git.io/JDH9G [00:54:18] [02mediawiki] 07github-actions[bot] labeled pull request 03#4680: Bump extensions/RemovePII from `41a11a3` to `8c799ed` - 13https://git.io/JDHSx [00:54:19] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-f35f446 [+0/-0/±1] 13https://git.io/JDH9n [00:54:21] [02miraheze/mediawiki] 07dependabot[bot] 0367c7eaf - Bump extensions/UniversalLanguageSelector from `357e68b` to `f35f446` [00:54:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#4679: Bump extensions/MatomoAnalytics from `4564cb1` to `0424557` - 13https://git.io/JDHS7 [00:54:24] [02mediawiki] 07dependabot[bot] opened pull request 03#4686: Bump extensions/UniversalLanguageSelector from `357e68b` to `f35f446` - 13https://git.io/JDH9c [00:54:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-f35f446 - 13https://git.io/vbL5b [00:54:27] [02mediawiki] 07github-actions[bot] labeled pull request 03#4681: Bump extensions/BlogPage from `f7526b8` to `a1b055b` - 13https://git.io/JDHSj [00:54:28] [02mediawiki] 07dependabot[bot] commented on pull request 03#4663: Bump extensions/UniversalLanguageSelector from `357e68b` to `a6c7690` - 13https://git.io/JDH9W [00:54:30] [02mediawiki] 07dependabot[bot] closed pull request 03#4663: Bump extensions/UniversalLanguageSelector from `357e68b` to `a6c7690` - 13https://git.io/JD2dJ [00:54:31] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-a6c7690 [00:54:33] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-a6c7690 - 13https://git.io/vbL5b [00:54:34] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SimpleTooltip-3030d31 [+0/-0/±1] 13https://git.io/JDH9l [00:54:36] [02miraheze/mediawiki] 07dependabot[bot] 0331d82c9 - Bump extensions/SimpleTooltip from `197be6f` to `3030d31` [00:54:37] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SimpleTooltip-3030d31 - 13https://git.io/vbL5b [00:54:39] [02mediawiki] 07dependabot[bot] opened pull request 03#4687: Bump extensions/SimpleTooltip from `197be6f` to `3030d31` - 13https://git.io/JDH94 [00:54:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Femiwiki-811ebbc [+0/-0/±1] 13https://git.io/JDH9R [00:54:42] [02miraheze/mediawiki] 07dependabot[bot] 03b5d93e6 - Bump skins/Femiwiki from `be075c6` to `811ebbc` [00:54:43] [02mediawiki] 07dependabot[bot] opened pull request 03#4688: Bump skins/Femiwiki from `be075c6` to `811ebbc` - 13https://git.io/JDH9E [00:54:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Femiwiki-811ebbc - 13https://git.io/vbL5b [00:54:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#4683: Bump extensions/FancyBoxThumbs from `ac84351` to `256e89c` - 13https://git.io/JDH9I [00:54:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#4682: Bump extensions/SocialProfile from `da33f37` to `952f5d4` - 13https://git.io/JDH9J [00:54:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AJAXPoll-1e37387 [+0/-0/±1] 13https://git.io/JDH9u [00:54:51] [02miraheze/mediawiki] 07dependabot[bot] 0362772d3 - Bump extensions/AJAXPoll from `4deb176` to `1e37387` [00:54:52] [02mediawiki] 07dependabot[bot] commented on pull request 03#4675: Bump skins/Femiwiki from `be075c6` to `8a99aef` - 13https://git.io/JDH9z [00:54:54] [02mediawiki] 07dependabot[bot] opened pull request 03#4689: Bump extensions/AJAXPoll from `4deb176` to `1e37387` - 13https://git.io/JDH9g [00:54:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AJAXPoll-1e37387 - 13https://git.io/vbL5b [00:54:57] [02mediawiki] 07dependabot[bot] closed pull request 03#4675: Bump skins/Femiwiki from `be075c6` to `8a99aef` - 13https://git.io/JDr30 [00:54:58] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/skins/Femiwiki-8a99aef - 13https://git.io/vbL5b [00:55:00] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/skins/Femiwiki-8a99aef [00:55:01] [02mediawiki] 07dependabot[bot] commented on pull request 03#4626: Bump extensions/AJAXPoll from `4deb176` to `c56884f` - 13https://git.io/JDH92 [00:55:03] [02mediawiki] 07dependabot[bot] closed pull request 03#4626: Bump extensions/AJAXPoll from `4deb176` to `c56884f` - 13https://git.io/JD27D [00:55:04] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/AJAXPoll-c56884f - 13https://git.io/vbL5b [00:55:06] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/AJAXPoll-c56884f [00:55:07] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Cargo-4878125 [+0/-0/±1] 13https://git.io/JDH9a [00:55:09] [02miraheze/mediawiki] 07dependabot[bot] 037ce20ce - Bump extensions/Cargo from `d100b8c` to `4878125` [00:55:11] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Cargo-4878125 - 13https://git.io/vbL5b [00:55:12] [02mediawiki] 07dependabot[bot] opened pull request 03#4690: Bump extensions/Cargo from `d100b8c` to `4878125` - 13https://git.io/JDH9w [00:55:14] [02mediawiki] 07dependabot[bot] commented on pull request 03#4662: Bump extensions/Cargo from `d100b8c` to `231829f` - 13https://git.io/JDH9o [00:55:15] [02mediawiki] 07dependabot[bot] closed pull request 03#4662: Bump extensions/Cargo from `d100b8c` to `231829f` - 13https://git.io/JD2dv [00:55:16] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Cargo-231829f - 13https://git.io/vbL5b [00:55:18] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Cargo-231829f [00:55:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#4685: Bump skins/Monaco from `90ccaf8` to `a07ea87` - 13https://git.io/JDH9G [00:55:21] [02mediawiki] 07github-actions[bot] labeled pull request 03#4685: Bump skins/Monaco from `90ccaf8` to `a07ea87` - 13https://git.io/JDH9G [00:55:22] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SimpleBlogPage-0bf4784 [+0/-0/±1] 13https://git.io/JDH9P [00:55:24] [02miraheze/mediawiki] 07dependabot[bot] 03caa050b - Bump extensions/SimpleBlogPage from `105256e` to `0bf4784` [00:55:25] [02mediawiki] 07dependabot[bot] opened pull request 03#4691: Bump extensions/SimpleBlogPage from `105256e` to `0bf4784` - 13https://git.io/JDH91 [00:55:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SimpleBlogPage-0bf4784 - 13https://git.io/vbL5b [00:55:29] [02mediawiki] 07github-actions[bot] labeled pull request 03#4684: Bump extensions/Translate from `0babb26` to `e189bf4` - 13https://git.io/JDH9O [00:55:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#4684: Bump extensions/Translate from `0babb26` to `e189bf4` - 13https://git.io/JDH9O [00:55:31] [02mediawiki] 07github-actions[bot] labeled pull request 03#4684: Bump extensions/Translate from `0babb26` to `e189bf4` - 13https://git.io/JDH9O [00:55:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#4686: Bump extensions/UniversalLanguageSelector from `357e68b` to `f35f446` - 13https://git.io/JDH9c [00:55:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#4687: Bump extensions/SimpleTooltip from `197be6f` to `3030d31` - 13https://git.io/JDH94 [00:55:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Wikibase-3087b60 [+0/-0/±1] 13https://git.io/JDH9D [00:55:38] [02miraheze/mediawiki] 07dependabot[bot] 03e63ba6d - Bump extensions/Wikibase from `1397f55` to `3087b60` [00:55:39] [02mediawiki] 07dependabot[bot] opened pull request 03#4692: Bump extensions/Wikibase from `1397f55` to `3087b60` - 13https://git.io/JDH9y [00:55:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Wikibase-3087b60 - 13https://git.io/vbL5b [00:55:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#4690: Bump extensions/Cargo from `d100b8c` to `4878125` - 13https://git.io/JDH9w [00:55:44] [02mediawiki] 07github-actions[bot] labeled pull request 03#4688: Bump skins/Femiwiki from `be075c6` to `811ebbc` - 13https://git.io/JDH9E [00:55:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#4690: Bump extensions/Cargo from `d100b8c` to `4878125` - 13https://git.io/JDH9w [00:55:47] [02mediawiki] 07dependabot[bot] commented on pull request 03#4609: Bump extensions/Wikibase from `1397f55` to `c433ffb` - 13https://git.io/JDH9S [00:55:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#4690: Bump extensions/Cargo from `d100b8c` to `4878125` - 13https://git.io/JDH9w [00:55:50] [02mediawiki] 07dependabot[bot] closed pull request 03#4609: Bump extensions/Wikibase from `1397f55` to `c433ffb` - 13https://git.io/JD27U [00:55:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikiForum-ca03daa [+0/-0/±1] 13https://git.io/JDH99 [00:55:53] [02miraheze/mediawiki] 07dependabot[bot] 034d51267 - Bump extensions/WikiForum from `e76b1cf` to `ca03daa` [00:55:54] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Wikibase-c433ffb - 13https://git.io/vbL5b [00:55:56] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Wikibase-c433ffb [00:55:57] [02mediawiki] 07dependabot[bot] opened pull request 03#4693: Bump extensions/WikiForum from `e76b1cf` to `ca03daa` - 13https://git.io/JDH9H [00:55:59] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikiForum-ca03daa - 13https://git.io/vbL5b [00:56:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#4689: Bump extensions/AJAXPoll from `4deb176` to `1e37387` - 13https://git.io/JDH9g [00:56:02] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Citizen-df7abc4 [+0/-0/±1] 13https://git.io/JDH9Q [00:56:03] [02miraheze/mediawiki] 07dependabot[bot] 03bcee7ab - Bump skins/Citizen from `ff8e8c4` to `df7abc4` [00:56:05] [02mediawiki] 07dependabot[bot] opened pull request 03#4694: Bump skins/Citizen from `ff8e8c4` to `df7abc4` - 13https://git.io/JDH97 [00:56:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Citizen-df7abc4 - 13https://git.io/vbL5b [00:56:07] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DynamicPageList3-b338ff7 [+0/-0/±1] 13https://git.io/JDH9d [00:56:09] [02miraheze/mediawiki] 07dependabot[bot] 03444e0d3 - Bump extensions/DynamicPageList3 from `db5e131` to `b338ff7` [00:56:11] [02mediawiki] 07dependabot[bot] opened pull request 03#4695: Bump extensions/DynamicPageList3 from `db5e131` to `b338ff7` - 13https://git.io/JDH9F [00:56:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DynamicPageList3-b338ff7 - 13https://git.io/vbL5b [00:56:14] [02mediawiki] 07dependabot[bot] commented on pull request 03#4631: Bump extensions/DynamicPageList3 from `db5e131` to `cf3e056` - 13https://git.io/JDH9N [00:56:15] [02mediawiki] 07dependabot[bot] closed pull request 03#4631: Bump extensions/DynamicPageList3 from `db5e131` to `cf3e056` - 13https://git.io/JD27N [00:56:17] [02mediawiki] 07github-actions[bot] labeled pull request 03#4691: Bump extensions/SimpleBlogPage from `105256e` to `0bf4784` - 13https://git.io/JDH91 [00:56:18] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/DynamicPageList3-cf3e056 - 13https://git.io/vbL5b [00:56:20] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/DynamicPageList3-cf3e056 [00:56:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Evelution-000ea9c [+0/-0/±1] 13https://git.io/JDH9j [00:56:23] [02miraheze/mediawiki] 07dependabot[bot] 032b33608 - Bump skins/Evelution from `2cc6564` to `000ea9c` [00:56:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Evelution-000ea9c - 13https://git.io/vbL5b [00:56:26] [02mediawiki] 07dependabot[bot] opened pull request 03#4696: Bump skins/Evelution from `2cc6564` to `000ea9c` - 13https://git.io/JDHHe [00:56:27] [02mediawiki] 07github-actions[bot] labeled pull request 03#4693: Bump extensions/WikiForum from `e76b1cf` to `ca03daa` - 13https://git.io/JDH9H [00:56:29] [02mediawiki] 07github-actions[bot] labeled pull request 03#4693: Bump extensions/WikiForum from `e76b1cf` to `ca03daa` - 13https://git.io/JDH9H [00:56:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageForms-1e4105c [+0/-0/±1] 13https://git.io/JDHHT [00:56:32] [02miraheze/mediawiki] 07dependabot[bot] 03f4fb999 - Bump extensions/PageForms from `5a376c5` to `1e4105c` [00:56:33] [02mediawiki] 07dependabot[bot] opened pull request 03#4697: Bump extensions/PageForms from `5a376c5` to `1e4105c` - 13https://git.io/JDHHk [00:56:35] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageForms-1e4105c - 13https://git.io/vbL5b [00:56:36] [02mediawiki] 07dependabot[bot] commented on pull request 03#4666: Bump extensions/PageForms from `5a376c5` to `e88f3c5` - 13https://git.io/JDHHI [00:56:38] [02mediawiki] 07dependabot[bot] closed pull request 03#4666: Bump extensions/PageForms from `5a376c5` to `e88f3c5` - 13https://git.io/JD2dm [00:56:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#4694: Bump skins/Citizen from `ff8e8c4` to `df7abc4` - 13https://git.io/JDH97 [00:56:41] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/PageForms-e88f3c5 [00:56:42] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/PageForms-e88f3c5 - 13https://git.io/vbL5b [00:56:44] [02mediawiki] 07github-actions[bot] labeled pull request 03#4694: Bump skins/Citizen from `ff8e8c4` to `df7abc4` - 13https://git.io/JDH97 [00:56:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#4692: Bump extensions/Wikibase from `1397f55` to `3087b60` - 13https://git.io/JDH9y [00:56:47] [02mediawiki] 07github-actions[bot] labeled pull request 03#4692: Bump extensions/Wikibase from `1397f55` to `3087b60` - 13https://git.io/JDH9y [00:56:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#4692: Bump extensions/Wikibase from `1397f55` to `3087b60` - 13https://git.io/JDH9y [00:56:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageSchemas-ce663ab [+0/-0/±1] 13https://git.io/JDHHt [00:56:51] [02miraheze/mediawiki] 07dependabot[bot] 038462f25 - Bump extensions/PageSchemas from `af3aa88` to `ce663ab` [00:56:53] [02mediawiki] 07dependabot[bot] opened pull request 03#4698: Bump extensions/PageSchemas from `af3aa88` to `ce663ab` - 13https://git.io/JDHHq [00:56:54] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageSchemas-ce663ab - 13https://git.io/vbL5b [00:56:56] [02mediawiki] 07github-actions[bot] labeled pull request 03#4695: Bump extensions/DynamicPageList3 from `db5e131` to `b338ff7` - 13https://git.io/JDH9F [00:56:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#4695: Bump extensions/DynamicPageList3 from `db5e131` to `b338ff7` - 13https://git.io/JDH9F [00:56:59] [02mediawiki] 07github-actions[bot] labeled pull request 03#4695: Bump extensions/DynamicPageList3 from `db5e131` to `b338ff7` - 13https://git.io/JDH9F [00:57:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#4696: Bump skins/Evelution from `2cc6564` to `000ea9c` - 13https://git.io/JDHHe [00:57:02] [02mediawiki] 07github-actions[bot] labeled pull request 03#4696: Bump skins/Evelution from `2cc6564` to `000ea9c` - 13https://git.io/JDHHe [00:57:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Gamepress-5ab4cf5 [+0/-0/±1] 13https://git.io/JDHHm [00:57:05] [02miraheze/mediawiki] 07dependabot[bot] 03964872e - Bump skins/Gamepress from `349349b` to `5ab4cf5` [00:57:06] [02mediawiki] 07dependabot[bot] opened pull request 03#4699: Bump skins/Gamepress from `349349b` to `5ab4cf5` - 13https://git.io/JDHHY [00:57:08] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Gamepress-5ab4cf5 - 13https://git.io/vbL5b [00:57:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/IncidentReporting-c31c24f [+0/-0/±1] 13https://git.io/JDHH3 [00:57:11] [02miraheze/mediawiki] 07dependabot[bot] 03e38f6d6 - Bump extensions/IncidentReporting from `583cede` to `c31c24f` [00:57:12] [02mediawiki] 07dependabot[bot] opened pull request 03#4700: Bump extensions/IncidentReporting from `583cede` to `c31c24f` - 13https://git.io/JDHHs [00:57:14] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/IncidentReporting-c31c24f - 13https://git.io/vbL5b [00:57:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DiscordNotifications-5ba66aa [+0/-0/±1] 13https://git.io/JDHHG [00:57:17] [02miraheze/mediawiki] 07dependabot[bot] 037021c77 - Bump extensions/DiscordNotifications from `1147510` to `5ba66aa` [00:57:18] [02mediawiki] 07dependabot[bot] opened pull request 03#4701: Bump extensions/DiscordNotifications from `1147510` to `5ba66aa` - 13https://git.io/JDHHZ [00:57:20] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DiscordNotifications-5ba66aa - 13https://git.io/vbL5b [00:57:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Preloader-2cf3c35 [+0/-0/±1] 13https://git.io/JDHHC [00:57:23] [02miraheze/mediawiki] 07dependabot[bot] 0380c983d - Bump extensions/Preloader from `e0d0343` to `2cf3c35` [00:57:24] [02mediawiki] 07dependabot[bot] opened pull request 03#4702: Bump extensions/Preloader from `e0d0343` to `2cf3c35` - 13https://git.io/JDHHW [00:57:26] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Preloader-2cf3c35 - 13https://git.io/vbL5b [00:57:27] [02mediawiki] 07github-actions[bot] labeled pull request 03#4697: Bump extensions/PageForms from `5a376c5` to `1e4105c` - 13https://git.io/JDHHk [00:57:29] [02mediawiki] 07github-actions[bot] labeled pull request 03#4697: Bump extensions/PageForms from `5a376c5` to `1e4105c` - 13https://git.io/JDHHk [00:57:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#4697: Bump extensions/PageForms from `5a376c5` to `1e4105c` - 13https://git.io/JDHHk [00:57:32] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AutoCreatePage-3871260 [+0/-0/±1] 13https://git.io/JDHHl [00:57:33] [02miraheze/mediawiki] 07dependabot[bot] 03bfcad50 - Bump extensions/AutoCreatePage from `1ee4e4b` to `3871260` [00:57:35] [02mediawiki] 07github-actions[bot] labeled pull request 03#4698: Bump extensions/PageSchemas from `af3aa88` to `ce663ab` - 13https://git.io/JDHHq [00:57:36] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AutoCreatePage-3871260 - 13https://git.io/vbL5b [00:57:38] [02mediawiki] 07dependabot[bot] opened pull request 03#4703: Bump extensions/AutoCreatePage from `1ee4e4b` to `3871260` - 13https://git.io/JDHH8 [00:57:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PortableInfobox-c934b05 [+0/-0/±1] 13https://git.io/JDHHB [00:57:41] [02miraheze/mediawiki] 07dependabot[bot] 0375d9f6a - Bump extensions/PortableInfobox from `25c6c90` to `c934b05` [00:57:42] [02mediawiki] 07dependabot[bot] opened pull request 03#4704: Bump extensions/PortableInfobox from `25c6c90` to `c934b05` - 13https://git.io/JDHHR [00:57:44] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PortableInfobox-c934b05 - 13https://git.io/vbL5b [00:57:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#4700: Bump extensions/IncidentReporting from `583cede` to `c31c24f` - 13https://git.io/JDHHs [00:57:47] [02mediawiki] 07github-actions[bot] labeled pull request 03#4699: Bump skins/Gamepress from `349349b` to `5ab4cf5` - 13https://git.io/JDHHY [00:57:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SlackNotifications-e8e6632 [+0/-0/±1] 13https://git.io/JDHH0 [00:57:50] [02miraheze/mediawiki] 07dependabot[bot] 0324813ed - Bump extensions/SlackNotifications from `3fe4d15` to `e8e6632` [00:57:51] [02mediawiki] 07dependabot[bot] opened pull request 03#4705: Bump extensions/SlackNotifications from `3fe4d15` to `e8e6632` - 13https://git.io/JDHHE [00:57:53] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SlackNotifications-e8e6632 - 13https://git.io/vbL5b [00:57:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ManageWiki-cdda89f [+0/-0/±1] 13https://git.io/JDHHg [00:57:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.49, 3.18, 3.26 [00:57:56] [02miraheze/mediawiki] 07dependabot[bot] 0385c9754 - Bump extensions/ManageWiki from `cdcbb97` to `cdda89f` [00:57:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ManageWiki-cdda89f - 13https://git.io/vbL5b [00:57:59] [02mediawiki] 07dependabot[bot] opened pull request 03#4706: Bump extensions/ManageWiki from `cdcbb97` to `cdda89f` - 13https://git.io/JDHH2 [00:58:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#4701: Bump extensions/DiscordNotifications from `1147510` to `5ba66aa` - 13https://git.io/JDHHZ [00:58:02] [02mediawiki] 07github-actions[bot] labeled pull request 03#4701: Bump extensions/DiscordNotifications from `1147510` to `5ba66aa` - 13https://git.io/JDHHZ [00:58:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#4703: Bump extensions/AutoCreatePage from `1ee4e4b` to `3871260` - 13https://git.io/JDHH8 [00:58:05] [02mediawiki] 07github-actions[bot] labeled pull request 03#4705: Bump extensions/SlackNotifications from `3fe4d15` to `e8e6632` - 13https://git.io/JDHHE [00:58:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#4706: Bump extensions/ManageWiki from `cdcbb97` to `cdda89f` - 13https://git.io/JDHH2 [00:58:08] [02mediawiki] 07github-actions[bot] labeled pull request 03#4704: Bump extensions/PortableInfobox from `25c6c90` to `c934b05` - 13https://git.io/JDHHR [00:58:09] [02mediawiki] 07github-actions[bot] labeled pull request 03#4706: Bump extensions/ManageWiki from `cdcbb97` to `cdda89f` - 13https://git.io/JDHH2 [00:58:11] [02mediawiki] 07github-actions[bot] labeled pull request 03#4704: Bump extensions/PortableInfobox from `25c6c90` to `c934b05` - 13https://git.io/JDHHR [00:58:12] [02mediawiki] 07github-actions[bot] labeled pull request 03#4706: Bump extensions/ManageWiki from `cdcbb97` to `cdda89f` - 13https://git.io/JDHH2 [00:58:14] [02mediawiki] 07github-actions[bot] labeled pull request 03#4704: Bump extensions/PortableInfobox from `25c6c90` to `c934b05` - 13https://git.io/JDHHR [00:58:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#4702: Bump extensions/Preloader from `e0d0343` to `2cf3c35` - 13https://git.io/JDHHW [01:01:08] * dmehus grabs 🍿 as he waits for CosmicAlpha to start merging the dependabot updates above :P [01:03:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.85, 19.03, 23.26 [01:07:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.04, 5.18, 5.96 [01:15:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.83, 16.77, 19.73 [01:15:28] dmehus: lol yeah I'll probably do that tomorrow. [01:17:06] 117 is all, not as bad as sometimes (there's been over 500 once) [01:21:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.17, 5.70, 5.50 [01:22:02] All except for 12 of the new PRs are my own patches, [01:22:53] Alot of build/tests stuff I've been doing to my own extensions. [01:23:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.48, 3.30 [01:24:33] I maintain 25 extensions now... I really don't know how much more extensions I even have the capacity to maintain. Every update I gain a few to keep them working it seems. [01:24:48] CosmicAlpha, heh [01:25:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.82, 4.68, 5.18 [01:29:02] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.03, 4.41, 4.97 [01:31:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.75, 3.82, 3.50 [01:33:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.91, 3.58 [01:34:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:35:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.40, 4.10, 3.69 [01:37:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.69, 3.59 [01:39:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:41:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.79, 3.63 [01:43:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.69, 3.61 [01:45:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.79, 3.66 [01:47:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.78, 3.80, 3.68 [01:49:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.44, 4.19, 3.82 [01:51:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.40, 19.82, 18.13 [01:51:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.94, 3.77 [01:53:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.24, 19.65, 18.35 [01:53:37] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.19, 5.22, 4.89 [01:55:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.37, 4.38, 3.95 [01:57:28] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.36, 5.34, 4.99 [01:59:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.62, 20.22, 18.86 [02:01:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.46, 19.61, 18.78 [02:01:19] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.23, 5.78, 5.29 [02:01:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.99, 3.93 [02:03:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 4.07, 3.96 [02:05:10] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.40, 5.64, 5.34 [02:07:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.06, 3.91, 3.94 [02:08:12] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.94, 1.89, 1.44 [02:09:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.62, 21.06, 19.68 [02:09:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:09:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.72, 4.74, 4.25 [02:10:12] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.29, 1.59, 1.38 [02:11:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.06, 5.83, 5.57 [02:13:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.30, 5.81, 5.58 [02:14:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:15:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.03, 5.29, 5.40 [02:17:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.75, 19.17, 19.31 [02:19:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.82, 6.27, 5.74 [02:19:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.76, 3.51, 3.95 [02:21:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.07, 21.68, 20.33 [02:22:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:23:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.43, 5.80, 5.73 [02:23:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.16, 19.37, 19.66 [02:23:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.98, 4.03 [02:24:15] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.40, 19.80, 16.70 [02:25:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.76, 3.61, 3.90 [02:26:15] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.88, 18.41, 16.58 [02:27:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:29:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.34, 6.35, 5.86 [02:29:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.42, 3.83, 3.90 [02:31:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.28, 21.12, 20.12 [02:31:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:33:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.41, 19.22, 19.53 [02:33:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.99, 3.95 [02:35:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.73, 5.39, 5.69 [02:35:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.85, 4.14, 4.00 [02:36:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:39:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.24, 3.71, 3.87 [02:41:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.46, 5.47, 5.48 [02:43:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.30, 5.53, 5.52 [02:43:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.88, 22.10, 20.46 [02:43:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.76, 4.06, 3.97 [02:45:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.51, 6.13, 5.76 [02:45:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.71, 3.50, 3.78 [02:47:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.16, 5.97, 5.76 [02:49:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.93, 19.82, 20.03 [02:51:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.82, 6.28, 5.85 [02:52:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:53:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.02, 5.91, 5.77 [02:53:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.55, 21.64, 20.84 [02:53:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.38, 3.53 [02:55:18] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.52, 22.66, 21.30 [02:55:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.40, 2.99, 3.37 [02:57:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:57:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.69, 19.96, 20.48 [02:59:18] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.44, 18.25, 19.77 [03:05:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.07, 5.48, 5.42 [03:07:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.07, 5.30, 5.36 [03:09:20] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.80, 1.43, 1.16 [03:11:19] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.11, 1.26, 1.13 [03:13:02] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.70, 3.98, 4.81 [03:50:48] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.15, 5.22, 4.70 [03:52:43] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.18, 4.89, 4.64 [04:23:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.58, 3.26, 2.76 [04:25:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.65, 3.08, 2.75 [05:00:03] .tell RhinosF1 ty for restoring the IRC/Discord relay for #miraheze-cvt-private [05:00:04] dmehus: I'll pass that on when RhinosF1 is around. [05:11:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:11:49] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.63, 5.20, 4.06 [05:12:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.19, 6.82, 5.75 [05:13:45] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.79, 4.72, 4.02 [05:14:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.99, 6.47, 5.75 [05:16:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:17:16] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.24, 1.94, 1.34 [05:18:46] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.71, 7.32, 6.03 [05:19:16] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.08, 1.64, 1.30 [05:21:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:22:21] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 8.00, 7.18, 5.94 [05:24:38] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.24, 6.62, 6.16 [05:26:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:30:08] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.87, 6.13, 6.00 [06:54:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:55:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.03, 5.14, 4.05 [06:57:02] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.84, 4.92, 4.10 [06:59:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:23:36] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.68, 4.85, 4.29 [07:29:22] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.75, 5.04, 4.64 [08:06:45] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.81, 6.96, 6.11 [08:08:44] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.52, 6.71, 6.12 [08:10:00] dmehus: if you wa [08:10:00] [tell] RhinosF1: 2021-12-20 - 05:00:04GMT tell RhinosF1 ty for restoring the IRC/Discord relay for #miraheze-cvt-private [08:10:11] Want rights for betawiki, let me know [08:23:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:28:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:23:16] Reception123: merge https://github.com/miraheze/puppet/pull/2163 & https://github.com/miraheze/ssl/pulls [09:23:17] [url] monitoring::ssl: allow domains pointing to test by RhinosF1 · Pull Request #2163 · miraheze/puppet · GitHub | github.com [09:23:18] [url] Pull requests · miraheze/ssl · GitHub | github.com [09:28:57] [02puppet] 07Reception123 closed pull request 03#2163: monitoring::ssl: allow domains pointing to test - 13https://git.io/JDS9a [09:28:59] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JD5Xn [09:29:00] [02miraheze/puppet] 07RhinosF1 0314d7e78 - monitoring::ssl: allow domains pointing to test (#2163) [09:29:05] [02ssl] 07Reception123 closed pull request 03#445: Remove expired oecumene.org - 13https://git.io/JDHTL [09:29:06] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JD5Xc [09:29:08] [02miraheze/ssl] 07Universal-Omega 032d72812 - Remove expired oecumene.org (#445) [09:29:11] [02ssl] 07Reception123 closed pull request 03#446: Update test1 and test2 redirects to point at beta.betaheze.org - 13https://git.io/JDHnB [09:29:12] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JD5XC [09:29:14] [02miraheze/ssl] 07Universal-Omega 03d51cfd8 - Update test1 and test2 redirects to point at beta.betaheze.org (#446) [09:35:06] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JD51t [09:35:08] [02miraheze/mw-config] 07RhinosF1 033e8fbe7 - LS: move realm specific config to separate files [09:35:09] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-3 - 13https://git.io/vbvb3 [09:35:49] [02mw-config] 07RhinosF1 opened pull request 03#4292: LS: move realm specific config to separate files - 13https://git.io/JD513 [09:36:50] miraheze/mw-config - RhinosF1 the build has errored. [09:36:51] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JD51B [09:36:53] [02miraheze/mw-config] 07RhinosF1 03cf4501c - Update LocalSettings.php [09:36:54] [02mw-config] 07RhinosF1 synchronize pull request 03#4292: LS: move realm specific config to separate files - 13https://git.io/JD513 [09:37:52] miraheze/mw-config - RhinosF1 the build has errored. [09:57:13] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+1/-0/±0] 13https://git.io/JD5SM [09:57:14] [02miraheze/mw-config] 07RhinosF1 032869169 - Create BetaSettings.php [09:57:16] [02mw-config] 07RhinosF1 synchronize pull request 03#4292: LS: move realm specific config to separate files - 13https://git.io/JD513 [09:58:17] miraheze/mw-config - RhinosF1 the build has errored. [09:58:40] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JD5Sh [09:58:42] [02miraheze/mw-config] 07RhinosF1 030db0194 - Update LocalSettings.php [09:58:43] [02mw-config] 07RhinosF1 synchronize pull request 03#4292: LS: move realm specific config to separate files - 13https://git.io/JD513 [09:59:09] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JD59k [09:59:11] [02miraheze/mw-config] 07github-actions 03e53ed42 - CI: lint code to MediaWiki standards [09:59:12] [02mw-config] 07github-actions[bot] synchronize pull request 03#4292: LS: move realm specific config to separate files - 13https://git.io/JD513 [09:59:45] miraheze/mw-config - RhinosF1 the build passed. [10:01:37] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+1/-0/±0] 13https://git.io/JD59K [10:01:39] [02miraheze/mw-config] 07RhinosF1 03acf4411 - Create ProductionSettings.php [10:01:40] [02mw-config] 07RhinosF1 synchronize pull request 03#4292: LS: move realm specific config to separate files - 13https://git.io/JD513 [10:02:39] miraheze/mw-config - RhinosF1 the build passed. [10:09:30] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JD5Qv [10:09:31] [02miraheze/mw-config] 07RhinosF1 0379501d4 - Update LocalSettings.php [10:09:33] [02mw-config] 07RhinosF1 synchronize pull request 03#4292: LS: move realm specific config to separate files - 13https://git.io/JD513 [10:10:33] miraheze/mw-config - RhinosF1 the build has errored. [10:10:33] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JD5QO [10:10:35] [02miraheze/mw-config] 07RhinosF1 032ce0bf0 - Update ProductionSettings.php [10:10:36] [02mw-config] 07RhinosF1 synchronize pull request 03#4292: LS: move realm specific config to separate files - 13https://git.io/JD513 [10:11:47] miraheze/mw-config - RhinosF1 the build has errored. [10:11:56] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JD5Qr [10:11:57] [02miraheze/mw-config] 07RhinosF1 03b044237 - Update LocalSettings.php [10:11:59] [02mw-config] 07RhinosF1 synchronize pull request 03#4292: LS: move realm specific config to separate files - 13https://git.io/JD513 [10:12:59] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JD5Qb [10:13:00] [02miraheze/mw-config] 07RhinosF1 034a7507a - Update Defines.php [10:13:02] [02mw-config] 07RhinosF1 synchronize pull request 03#4270: Experimental: update for 1.37 - 13https://git.io/JDvis [10:13:04] miraheze/mw-config - RhinosF1 the build passed. [10:13:53] RECOVERY - beta.betaheze.org - reverse DNS on sslhost is OK: SSL OK - beta.betaheze.org reverse DNS resolves to test3.miraheze.org - NS RECORDS OK [10:14:04] miraheze/mw-config - RhinosF1 the build passed. [10:14:19] [02mw-config] 07RhinosF1 closed pull request 03#4270: Experimental: update for 1.37 - 13https://git.io/JDvis [10:14:20] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 - 13https://git.io/vbvb3 [10:14:22] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 [10:14:23] RECOVERY - betaheze.org - reverse DNS on sslhost is OK: SSL OK - betaheze.org reverse DNS resolves to test3.miraheze.org - NS RECORDS OK [10:14:24] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JD57I [10:14:25] [02miraheze/mw-config] 07RhinosF1 031d43b1b - Experimental: update for 1.37 (#4270) [10:14:29] [02mw-config] 07supertassu reviewed pull request 03#4292 commit - 13https://git.io/JD57t [10:14:30] [02mw-config] 07supertassu reviewed pull request 03#4292 commit - 13https://git.io/JD57q [10:15:25] miraheze/mw-config - RhinosF1 the build passed. [10:15:44] !log [@test3] starting deploy of {'config': True} to skip [10:15:45] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [10:15:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:15:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:18:05] !log [@mw11] starting deploy of {'config': True} to ovlon [10:18:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:18:15] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 9s [10:18:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:20:55] !log refreshImageMetadata on experimental wikis [10:21:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:39:27] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JD5NP [10:39:28] [02miraheze/mw-config] 07RhinosF1 038781b94 - Update BetaSettings.php [10:39:30] [02mw-config] 07RhinosF1 synchronize pull request 03#4292: LS: move realm specific config to separate files - 13https://git.io/JD513 [10:39:44] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JD5NM [10:39:46] [02miraheze/mw-config] 07RhinosF1 03d0094d9 - Update ProductionSettings.php [10:39:47] [02mw-config] 07RhinosF1 synchronize pull request 03#4292: LS: move realm specific config to separate files - 13https://git.io/JD513 [10:39:57] [02mw-config] 07RhinosF1 reviewed pull request 03#4292 commit - 13https://git.io/JD5N5 [10:40:01] [02mw-config] 07RhinosF1 reviewed pull request 03#4292 commit - 13https://git.io/JD5Nd [10:40:29] miraheze/mw-config - RhinosF1 the build has errored. [10:40:41] miraheze/mw-config - RhinosF1 the build has errored. [10:41:23] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JD5Ak [10:41:25] [02miraheze/mw-config] 07RhinosF1 03e5dabd6 - Update BetaSettings.php [10:41:26] [02mw-config] 07RhinosF1 synchronize pull request 03#4292: LS: move realm specific config to separate files - 13https://git.io/JD513 [10:41:38] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/JD5At [10:41:39] [02miraheze/mw-config] 07RhinosF1 0345a3cb3 - Update ProductionSettings.php [10:41:41] [02mw-config] 07RhinosF1 synchronize pull request 03#4292: LS: move realm specific config to separate files - 13https://git.io/JD513 [10:42:25] miraheze/mw-config - RhinosF1 the build has errored. [10:42:35] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.77, 5.50, 4.77 [10:42:43] miraheze/mw-config - RhinosF1 the build passed. [10:43:37] majavah: done [10:48:18] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.21, 6.31, 5.29 [10:48:49] gluster load might be me [10:49:16] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 32.05, 22.83, 18.11 [10:49:59] @everyone :video_game: • Get Discord Nitro for Free from Steam Store Free 3 months Discord Nitro :clock630: • Personalize your profile, screen share in HD, upgrade your emojis, and more. :gem: • Click to get Nitro: https://discord-gifft.com/welcome :Works only with prime go or rust or pubg [10:50:00] [url] 3 months of Discord Nitro free from STEAM | discord-gifft.com [10:51:12] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.94, 22.34, 18.45 [10:53:09] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.64, 25.03, 19.95 [10:55:04] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.51, 3.82, 2.07 [10:55:06] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.80, 21.58, 19.30 [10:58:59] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.89, 20.06, 19.21 [10:59:04] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.38, 1.84, 1.64 [11:01:04] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.17, 1.28, 1.46 [11:03:33] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.86, 5.72, 5.81 [11:06:42] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.12, 20.88, 19.79 [11:08:39] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.30, 18.17, 18.93 [11:11:11] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.82, 6.18, 5.95 [11:14:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.39, 24.61, 21.25 [11:16:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.80, 23.82, 21.33 [11:20:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.46, 22.67, 21.28 [11:22:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:26:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.34, 23.50, 22.37 [11:28:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.21, 24.02, 22.67 [11:30:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.97, 22.35, 22.22 [11:34:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.59, 5.16, 5.89 [11:37:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:40:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.70, 17.24, 19.66 [11:40:53] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.00, 6.22, 5.65 [11:42:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:44:50] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.62, 6.38, 5.85 [11:44:55] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.10, 5.26, 5.42 [11:47:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:48:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.62, 5.41, 5.43 [11:50:55] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.90, 6.47, 5.82 [11:54:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.19, 5.21, 5.50 [12:00:55] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.01, 5.47, 5.52 [12:02:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.62, 4.94, 5.31 [12:08:35] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/JDdsy [12:08:37] [02miraheze/landing] 07translatewiki 03b948c8b - Localisation updates from https://translatewiki.net. [12:08:38] [url] Main page - translatewiki.net | translatewiki.net [12:08:38] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±27] 13https://git.io/JDdsS [12:08:40] [02miraheze/CreateWiki] 07translatewiki 038daba69 - Localisation updates from https://translatewiki.net. [12:08:41] [url] Main page - translatewiki.net | translatewiki.net [12:08:41] [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JDds9 [12:08:43] [02miraheze/DataDump] 07translatewiki 03ed580a9 - Localisation updates from https://translatewiki.net. [12:08:44] [url] Main page - translatewiki.net | translatewiki.net [12:08:44] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/JDdsQ [12:08:46] [02miraheze/MirahezeMagic] 07translatewiki 03b7f0904 - Localisation updates from https://translatewiki.net. [12:08:47] [url] Main page - translatewiki.net | translatewiki.net [12:08:54] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.34, 4.74, 5.09 [12:11:27] miraheze/landing - translatewiki the build passed. [12:16:58] miraheze/DataDump - translatewiki the build passed. [12:17:13] miraheze/CreateWiki - translatewiki the build passed. [12:21:12] miraheze/MirahezeMagic - translatewiki the build passed. [12:26:47] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.46, 6.29, 5.62 [12:27:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.89, 3.62, 3.09 [12:28:11] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.23, 7.06, 6.28 [12:29:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.70, 3.71, 3.17 [12:30:09] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.42, 6.50, 6.18 [12:30:47] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.89, 6.27, 5.79 [12:31:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.46, 3.14, 3.02 [12:56:25] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 14.66, 8.49, 5.57 [12:58:22] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.32, 7.37, 5.52 [13:00:22] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.23, 6.50, 5.41 [13:01:14] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 20.31, 10.93, 6.75 [13:02:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:07:11] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.86, 6.55, 6.18 [13:07:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:28:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.63, 5.72, 4.80 [13:29:20] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.16, 18.62, 17.74 [13:31:17] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.83, 19.19, 18.07 [13:31:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:34:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.64, 5.57, 5.15 [13:36:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:36:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.26, 3.05 [13:36:55] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.84, 4.64, 4.86 [13:40:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.23, 5.00, 4.93 [13:41:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:42:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.96, 3.25, 3.11 [13:42:54] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.59, 4.76, 4.86 [13:46:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:49:49] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.38, 7.67, 5.92 [13:50:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:50:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.63, 3.98, 3.45 [13:51:48] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.43, 7.26, 5.99 [13:52:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.83, 3.57, 3.37 [13:52:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.59, 7.16, 5.87 [13:53:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.19, 21.17, 19.81 [13:53:46] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.13, 6.56, 5.89 [13:56:56] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.10, 1.62, 1.11 [13:57:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.51, 22.75, 20.67 [13:58:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.39, 5.96, 5.84 [13:58:56] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.24, 1.45, 1.11 [13:59:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.45, 21.62, 20.53 [13:59:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.77, 3.33, 3.36 [14:00:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:00:06] PROBLEM - with.cpt-ra.bid - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:00:47] PROBLEM - you.r-fit.cc - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.query(root_domain, 'NS') File "/usr/lib/python3/d [14:00:47] ackages/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query r-fit.cc. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out [14:00:47] rver 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [14:02:28] PROBLEM - with.cpt-ra.bid - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - with.cpt-ra.bid All nameservers failed to answer the query. [14:02:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.77, 6.53, 6.06 [14:03:16] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.41, 22.15, 20.87 [14:05:13] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.66, 22.22, 21.08 [14:07:59] PROBLEM - you.r-fit.cc - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - you.r-fit.cc All nameservers failed to answer the query. [14:09:39] PROBLEM - with.cpt-ra.bid - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.query(hostnam [14:09:39] ')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.003830432891846 seconds [14:13:59] RECOVERY - with.cpt-ra.bid - LetsEncrypt on sslhost is OK: OK - Certificate 'with.cpt-ra.bid' will expire on Fri 14 Jan 2022 03:45:14 GMT +0000. [14:14:42] RECOVERY - you.r-fit.cc - reverse DNS on sslhost is OK: SSL OK - you.r-fit.cc reverse DNS resolves to cp21.miraheze.org - CNAME OK [14:16:21] RECOVERY - with.cpt-ra.bid - reverse DNS on sslhost is OK: SSL OK - with.cpt-ra.bid reverse DNS resolves to cp20.miraheze.org - CNAME OK [14:16:52] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.77, 19.42, 20.33 [14:19:38] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.93, 6.13, 5.60 [14:20:42] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.01, 21.53, 20.99 [14:21:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:21:25] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.25, 5.75, 5.12 [14:23:22] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.86, 5.41, 5.07 [14:25:33] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.01, 7.73, 6.47 [14:26:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.81, 22.75, 21.70 [14:27:35] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.25, 6.58, 5.70 [14:29:29] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.19, 7.50, 6.65 [14:29:34] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.37, 6.96, 5.95 [14:30:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.89, 21.99, 21.69 [14:32:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.41, 23.24, 22.19 [14:33:31] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.52, 6.40, 5.99 [14:34:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.45, 22.93, 22.23 [14:35:24] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.19, 6.61, 6.51 [14:37:56] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.31, 6.91, 5.93 [14:39:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.71, 6.83, 6.02 [14:41:08] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.53, 3.99, 3.45 [14:41:50] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.90, 6.18, 5.88 [14:42:08] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.51, 7.29, 6.24 [14:43:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.04, 3.75, 3.44 [14:46:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:46:06] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.03, 6.35, 6.11 [14:46:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.82, 22.68, 21.98 [14:48:37] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.18, 6.80, 6.25 [14:50:35] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.06, 6.98, 6.39 [14:50:43] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.32, 4.30, 3.72 [14:51:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:56:27] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.50, 6.65, 6.50 [14:59:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.85, 7.54, 6.53 [15:01:24] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.83, 6.41, 6.22 [15:02:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.71, 22.67, 23.77 [15:06:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:07:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:07:49] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.85, 7.14, 6.53 [15:07:51] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.13, 7.52, 6.76 [15:08:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.26, 22.48, 23.18 [15:09:49] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.64, 7.45, 6.83 [15:10:03] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.63, 7.18, 6.50 [15:10:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.34, 21.40, 22.69 [15:11:48] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.60, 7.89, 7.06 [15:12:01] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.19, 6.78, 6.44 [15:12:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:13:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:13:45] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.69, 6.49, 6.46 [15:15:57] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.43, 7.53, 6.82 [15:16:10] PROBLEM - www.aarthal.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'aarthal.com' expires in 15 day(s) (Wed 05 Jan 2022 15:07:26 GMT +0000). [15:16:16] PROBLEM - aarthal.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'aarthal.com' expires in 15 day(s) (Wed 05 Jan 2022 15:07:26 GMT +0000). [15:16:19] !log superwiki: > delete from page where page_id=388; [15:16:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:17:42] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.30, 7.16, 7.09 [15:17:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.97, 3.71, 3.93 [15:17:56] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.91, 7.03, 6.71 [15:18:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:18:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.75, 23.69, 23.16 [15:21:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.75, 4.02, 3.97 [15:22:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.59, 23.12, 23.13 [15:23:12] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.58, 6.90, 6.37 [15:23:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.16, 7.39, 7.20 [15:24:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 32.09, 26.25, 24.23 [15:25:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:25:11] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.75, 6.69, 6.34 [15:25:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.99, 7.23, 7.13 [15:25:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.81, 3.92 [15:26:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.90, 23.80, 23.59 [15:27:49] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.31, 6.58, 6.80 [15:27:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.66, 4.10, 4.01 [15:30:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.50, 5.52, 5.94 [15:31:04] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.19, 2.57, 1.59 [15:31:30] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.11, 6.36, 6.78 [15:33:04] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.65, 1.82, 1.43 [15:35:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:35:04] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.55, 1.41, 1.33 [15:36:18] PROBLEM - gs.sidem.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'gs.sidem.wiki' expires in 15 day(s) (Wed 05 Jan 2022 15:31:43 GMT +0000). [15:36:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.71, 6.76, 6.32 [15:37:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.84, 3.98 [15:38:24] PROBLEM - www.sidem.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.sidem.wiki' expires in 15 day(s) (Wed 05 Jan 2022 15:33:38 GMT +0000). [15:38:34] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.97, 6.59, 6.32 [15:38:55] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.37, 4.31, 5.10 [15:40:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:40:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 37.13, 25.15, 22.91 [15:41:40] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.23, 6.69, 6.29 [15:41:59] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [15:42:20] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.25, 6.57, 6.29 [15:42:21] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:42:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 7.19, 8.15, 7.13 [15:42:36] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.08, 7.53, 6.94 [15:42:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.15, 8.83, 6.87 [15:43:37] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.63, 6.09, 6.13 [15:43:58] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:44:00] PROBLEM - wikimas.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikimas.kr' expires in 15 day(s) (Wed 05 Jan 2022 15:40:34 GMT +0000). [15:44:17] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.265 second response time [15:44:19] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.57, 6.41, 6.25 [15:44:26] PROBLEM - tot.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tot.wiki' expires in 15 day(s) (Wed 05 Jan 2022 15:41:51 GMT +0000). [15:44:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.08, 7.66, 7.10 [15:44:35] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.97, 6.77, 6.75 [15:45:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.01, 2.74, 3.40 [15:46:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.54, 23.49, 23.20 [15:48:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 33.41, 26.37, 24.21 [15:48:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.46, 8.60, 7.49 [15:50:29] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.04, 7.20, 6.95 [15:50:33] [02mw-config] 07JohnFLewis commented on pull request 03#4292: LS: move realm specific config to separate files - 13https://git.io/JDF3U [15:50:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.09, 7.68, 7.30 [15:50:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.70, 6.83, 6.55 [15:51:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.46, 6.93, 6.57 [15:51:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.47, 3.51 [15:52:50] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.01, 6.33, 6.41 [15:53:21] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.32, 6.15, 6.34 [15:53:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.27, 3.14, 3.40 [15:54:26] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.79, 5.86, 6.50 [15:54:35] !log delete page_id=407 on superwiki [15:55:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:55:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:56:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.93, 5.65, 6.53 [15:57:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.10, 3.53, 3.53 [15:57:52] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.33, 3.02, 1.95 [15:58:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.62, 23.19, 23.83 [15:59:09] !log delete page_id=3 on beta wiki after testing cause of previous dangling pages [15:59:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:59:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:00:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.48, 23.49, 23.84 [16:01:44] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.01, 1.95, 1.75 [16:03:40] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.78, 1.65, 1.67 [16:03:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.33, 3.00, 3.34 [16:04:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:04:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.56, 23.88, 23.87 [16:05:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.15, 7.14, 6.60 [16:07:38] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.82, 6.46, 6.40 [16:08:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.01, 23.42, 23.58 [16:10:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.29, 23.06, 23.41 [16:11:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.57, 3.47, 3.37 [16:12:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.38, 25.06, 24.13 [16:13:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:13:47] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.28, 7.06, 6.46 [16:13:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.53, 3.40 [16:14:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.50, 23.19, 23.54 [16:15:45] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.81, 6.63, 6.38 [16:15:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.86, 3.30, 3.33 [16:18:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:19:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.82, 3.56 [16:21:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.64, 4.54, 3.85 [16:24:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.77, 16.83, 20.05 [16:24:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.64, 4.65, 5.94 [16:36:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.12, 4.97, 5.28 [16:38:10] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.27, 20.85, 20.09 [16:38:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.02, 4.90, 5.21 [16:39:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.87, 3.98 [16:40:06] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.51, 20.00, 19.89 [16:40:54] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.36, 4.65, 5.07 [16:43:57] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.19, 22.48, 20.95 [16:44:55] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.85, 5.81, 5.44 [16:46:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.35, 5.36, 5.33 [16:47:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.73, 3.92, 3.85 [16:48:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.15, 5.78, 5.50 [16:53:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.89, 3.88 [16:56:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.52, 5.95, 5.84 [17:04:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:04:55] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.35, 7.07, 6.27 [17:06:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.50, 5.88, 5.93 [17:09:45] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.62, 6.84, 5.99 [17:11:44] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.92, 7.21, 6.21 [17:13:01] found some problems with the skin Flatbox [17:13:10] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.67, 21.96, 21.24 [17:13:14] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.95, 7.34, 6.00 [17:13:21] which are? [17:13:48] 1. it tries to load a font from upload.wikimedia.org (non-whitelisted domain it seems) 2. it tries to load a file called sidebar.js, but it can't find it so returns a 404 [17:14:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.43, 5.91, 5.75 [17:15:13] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.45, 7.09, 6.06 [17:15:41] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.31, 6.98, 6.41 [17:17:08] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.67, 7.18, 6.26 [17:17:11] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.62, 5.75, 5.70 [17:17:40] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.02, 7.33, 6.60 [17:19:06] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.49, 8.14, 6.73 [17:19:38] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.19, 7.36, 6.71 [17:21:17] [02mediawiki] 07RhinosF1 closed pull request 03#4703: Bump extensions/AutoCreatePage from `1ee4e4b` to `3871260` - 13https://git.io/JDHH8 [17:21:18] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/AutoCreatePage-3871260 [17:21:20] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/AutoCreatePage-3871260 - 13https://git.io/vbL5b [17:21:21] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JDFaU [17:21:23] [02miraheze/mediawiki] 07dependabot[bot] 03ab418ca - Bump extensions/AutoCreatePage from `1ee4e4b` to `3871260` (#4703) [17:21:28] [02mediawiki] 07RhinosF1 closed pull request 03#4702: Bump extensions/Preloader from `e0d0343` to `2cf3c35` - 13https://git.io/JDHHW [17:21:30] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JDFaT [17:21:31] [02miraheze/mediawiki] 07dependabot[bot] 03ad5f5d8 - Bump extensions/Preloader from `e0d0343` to `2cf3c35` (#4702) [17:21:33] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Preloader-2cf3c35 [17:21:34] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/Preloader-2cf3c35 - 13https://git.io/vbL5b [17:21:37] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.87, 8.00, 6.98 [17:21:40] [02mediawiki] 07RhinosF1 closed pull request 03#4691: Bump extensions/SimpleBlogPage from `105256e` to `0bf4784` - 13https://git.io/JDH91 [17:21:41] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/SimpleBlogPage-0bf4784 - 13https://git.io/vbL5b [17:21:43] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JDFaL [17:21:44] [02miraheze/mediawiki] 07dependabot[bot] 033febb9b - Bump extensions/SimpleBlogPage from `105256e` to `0bf4784` (#4691) [17:21:46] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/SimpleBlogPage-0bf4784 [17:21:47] [02mediawiki] 07RhinosF1 closed pull request 03#4687: Bump extensions/SimpleTooltip from `197be6f` to `3030d31` - 13https://git.io/JDH94 [17:21:49] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JDFaq [17:21:50] [02miraheze/mediawiki] 07dependabot[bot] 0378e04dd - Bump extensions/SimpleTooltip from `197be6f` to `3030d31` (#4687) [17:21:52] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/SimpleTooltip-3030d31 - 13https://git.io/vbL5b [17:21:53] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/SimpleTooltip-3030d31 [17:21:56] [02mediawiki] 07RhinosF1 closed pull request 03#4683: Bump extensions/FancyBoxThumbs from `ac84351` to `256e89c` - 13https://git.io/JDH9I [17:21:58] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JDFa3 [17:21:59] [02miraheze/mediawiki] 07dependabot[bot] 037991c7c - Bump extensions/FancyBoxThumbs from `ac84351` to `256e89c` (#4683) [17:22:01] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/FancyBoxThumbs-256e89c - 13https://git.io/vbL5b [17:22:02] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/FancyBoxThumbs-256e89c [17:23:06] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.75, 7.85, 6.57 [17:23:35] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.21, 7.70, 7.00 [17:23:50] !log [rhinos@test3] starting deploy of {'world': True} to skip [17:23:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:24:04] !log [rhinos@mw11] starting deploy of {'world': True} to all [17:24:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:24:26] [02mw-config] 07Universal-Omega commented on pull request 03#4292: LS: move realm specific config to separate files - 13https://git.io/JDFaP [17:24:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 7.31, 8.35, 6.98 [17:24:50] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.48, 23.37, 23.00 [17:25:05] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.24, 7.72, 6.69 [17:26:00] CosmicAlpha: my aim is to get it so when we have more than 1 beta wiki, we don't have to copy all the config out again [17:26:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.42, 7.86, 6.97 [17:26:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.00, 5.02, 5.80 [17:26:55] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.24, 6.92, 6.89 [17:28:21] RhinosF1: well I still think we don't need a new ProductionSettings then, even then, listing all betawikis as an array in LS to use to include the file I don't necessarily like either. If all are going to be on betaheze.org, then we can rather check for that domain also. Either way I still think another file might not be necessary right now. [17:28:53] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.36, 6.34, 6.68 [17:28:56] CosmicAlpha: we can't use url or maint scripts wouldn't work [17:29:02] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.93, 6.67, 6.49 [17:29:31] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.22, 7.49, 7.06 [17:29:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.68, 3.08, 3.39 [17:29:56] !log [rhinos@test3] finished deploy of {'world': True} to skip - SUCCESS in 366s [17:30:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:30:21] RhinosF1: why not? Can it not check $wgServer, which is the same in maint scripts. [17:30:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.53, 7.79, 7.09 [17:32:54] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.12, 4.05, 5.09 [17:33:22] !log [rhinos@mw11] finished deploy of {'world': True} to all - SUCCESS in 558s [17:33:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:33:28] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.58, 7.86, 7.37 [17:34:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:34:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.64, 7.73, 7.27 [17:36:35] CosmicAlpha: $wgServer should work [17:36:54] How do we set config for a group though [17:38:57] $wgServer is set by WikiInitialise to wiki.miraheze.org as a default, we just need to setup betaheze.org in the setVariables config I think, though not certain. [17:40:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.03, 7.65, 7.29 [17:41:21] CosmicAlpha: eh? [17:41:22] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.14, 7.06, 7.07 [17:41:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:41:54] https://github.com/miraheze/CreateWiki/blob/9a3be23a28ab0e52c4e0963c01a950422e91afc9/includes/WikiInitialise.php#L57 [17:41:55] [url] CreateWiki/WikiInitialise.php at 9a3be23a28ab0e52c4e0963c01a950422e91afc9 · miraheze/CreateWiki · GitHub | github.com [17:42:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.92, 7.17, 7.17 [17:43:21] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.17, 6.68, 6.93 [17:43:28] CosmicAlpha: how's that gonna help us work out whether we are in beta or prod [17:43:46] And set config based on that [17:45:27] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.94, 6.73, 6.52 [17:46:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:46:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.84, 3.67, 3.43 [17:48:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.89, 6.13, 6.71 [17:49:17] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.29, 6.50, 6.79 [17:50:30] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.17, 3.80, 3.51 [17:50:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.12, 6.55, 5.62 [17:51:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:52:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.70, 3.52 [17:52:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.24, 23.01, 21.83 [17:53:14] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.96, 8.01, 7.27 [17:53:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.03, 6.53, 6.54 [17:55:58] RhinosF1: rather than listing each beta wiki we check if wgServer is on betaheze.org if all are going to be there is what I'm saying [17:56:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.91, 4.19, 3.72 [17:56:48] CosmicAlpha: yeah we can do that but we still need to decide where to put the variables [17:56:59] That doesn't eliminate two files [17:58:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.88, 3.66 [17:58:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.75, 22.23, 22.12 [17:59:19] RhinosF1: no but why do we need a production file, when it can just be in LS, also some sort of support might be able to be added to SiteConfiguration to support everything in LS, and support doing betaheze in replace of wikidb to set for all wikis on the betaheze domain. [17:59:52] Error contacting the Parsoid/RESTBase server (HTTP 404 [17:59:53] CosmicAlpha: the latter would be very good [18:00:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.46, 4.68, 3.97 [18:00:13] so omega was able to load visual edtior but when i try loading VE it dont work [18:00:20] Cocopuff2018: I can't reproduce the issue you mentioned on tsskz on the page you mentioned. [18:00:30] I don't mind an if statement at the top and putting the variables there but the SiteConfiguation would be nicer [18:00:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.75, 5.83, 6.00 [18:00:59] @cosmicAlpha so what we do to fix it, its happening on multiple pages [18:01:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:01:11] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.38, 7.50, 7.74 [18:05:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.79, 3.68, 3.74 [18:06:51] @CosmicAlpha it happens on every page [18:07:39] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.83, 7.13, 6.28 [18:07:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 4.03, 3.87 [18:08:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:08:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.13, 18.43, 20.16 [18:08:55] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.56, 3.95, 5.00 [18:11:36] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.99, 7.37, 6.60 [18:12:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.98, 20.36, 20.60 [18:13:11] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.30, 5.70, 6.59 [18:16:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.72, 19.32, 20.17 [18:19:05] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.20, 6.95, 6.37 [18:19:05] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.97, 6.66, 6.40 [18:19:31] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.25, 7.93, 7.12 [18:20:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.52, 20.91, 20.59 [18:20:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.72, 6.74, 6.23 [18:21:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.13, 7.74, 6.84 [18:21:03] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.73, 6.57, 6.31 [18:22:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.90, 6.27, 6.10 [18:22:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.08, 22.57, 21.22 [18:23:00] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.00, 7.39, 6.82 [18:23:28] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.42, 6.02, 5.29 [18:23:28] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.91, 7.40, 7.08 [18:24:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.65, 21.37, 20.96 [18:25:22] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.42, 5.26, 5.09 [18:25:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.92, 3.58, 3.94 [18:26:54] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 7.90, 8.01, 7.22 [18:27:17] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.18, 5.72, 5.26 [18:28:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:28:52] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.66, 7.26, 7.03 [18:29:11] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.35, 5.46, 5.22 [18:29:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:29:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.85, 3.89, 3.95 [18:30:02] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.31, 6.57, 6.21 [18:30:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.89, 23.34, 21.76 [18:31:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.94, 3.97 [18:32:01] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.91, 6.35, 6.18 [18:32:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.15, 22.46, 21.65 [18:32:46] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.12, 6.34, 6.69 [18:34:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:34:54] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.12, 4.51, 4.92 [18:37:18] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.03, 5.74, 6.54 [18:38:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.49, 18.93, 20.23 [18:46:13] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JDFSK [18:46:15] [02miraheze/CreateWiki] 07Universal-Omega 037825713 - Add betaheze to SiteConfiguration [18:46:16] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [18:46:19] [02CreateWiki] 07Universal-Omega opened pull request 03#268: Add betaheze to SiteConfiguration - 13https://git.io/JDFSi [18:46:54] RhinosF1: ^ not really sure if it'll work, and might need to be a bit different but could test it I guess. [18:47:48] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.89, 6.53, 6.05 [18:48:50] CosmicAlpha: can do in morning [18:51:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.86, 3.77 [18:52:00] CosmicAlpha: Don't we have a policy of not having Miraheze specific code in extensions? [18:52:17] miraheze/CreateWiki - Universal-Omega the build passed. [18:53:44] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.44, 6.13, 6.10 [18:53:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.79, 3.75 [18:59:52] * majavah points at the extensions named MirahezeSomething [19:03:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.45, 3.55 [19:04:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:04:08] majavah: MirahezeMagic, right? [19:04:36] JohnLewis: yes, that's why I mentioned "might need to be a bit different" should've clarified. I don't want this PR merged just yet, but wanted to test it before moving it somewhere else. I'll mention on PR also. Sorry for not clarifying that, as I did make it seem like this should be merged which was wrong. [19:05:04] [02CreateWiki] 07Universal-Omega edited pull request 03#268: [DNM] Add betaheze to SiteConfiguration - 13https://git.io/JDFSi [19:05:16] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.36, 5.67, 5.07 [19:05:40] [02CreateWiki] 07Universal-Omega edited pull request 03#268: [DNM] Add betaheze to SiteConfiguration - 13https://git.io/JDFSi [19:05:56] You can do it the same was at extensions, and then they’ll be tagged, so you can do ‘beta’ => $setting in LS [19:06:11] Via site configuration wiki tags [19:07:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.66, 3.63 [19:08:01] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.50, 6.89, 6.31 [19:08:29] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.42, 8.13, 6.65 [19:08:53] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.64, 21.05, 19.74 [19:09:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:09:05] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.81, 5.81, 5.26 [19:10:30] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.86, 7.42, 6.75 [19:10:49] JohnLewis: I'm not really sure what you mean by "the same was at extensions", as I thought I did this currently the same way extensions were done, granted yes, much more Miraheze specific which is why I wanted to figure something else out but didn't know what yet so wanted to see if this would work. [19:10:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.60, 20.37, 19.65 [19:11:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:11:55] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.02, 6.31, 6.22 [19:12:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.35, 7.89, 6.92 [19:12:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.28, 1.83, 1.27 [19:14:27] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.05, 6.75, 6.64 [19:14:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.59, 2.28, 1.50 [19:14:54] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.74, 5.04, 5.09 [19:15:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.40, 3.79, 3.65 [19:16:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:16:21] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.02, 5.63, 6.25 [19:16:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.21, 1.89, 1.45 [19:17:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.27, 3.34, 3.51 [19:18:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 7.69, 3.79, 2.17 [19:18:56] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.41, 7.08, 5.85 [19:19:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.83, 4.68, 3.97 [19:20:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:20:31] Wiki tags are set in LS, not CreateWiki. You just need to add the logic to allow the ‘beta’ tag to be added as a wiki tag. Which can be done in numerous way, either from just hard coding a ‘beta’ config variable that can be set, or adding a new options in CreateWiki to flag a wiki as a ‘beta’ wiki like is done with experimental [19:20:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.18, 25.77, 22.39 [19:21:18] If we can add wiki tags like we have for extensions then that would be ideal [19:21:33] And we can just replace betawiki with the tag [19:21:46] JohnLewis: how we going to add wikis to the tag, just in LS [19:22:15] That depends how it’s implemented [19:22:20] beta isn't something that's just a tickbox and most should work [19:22:28] So I don't think ManageWiki is the best place [19:23:33] Experimental the way it's been done is supposed to be to test impact and nothing in there is going to completely break anything unless we've misjudged it [19:23:35] Then yes, via LS [19:23:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.78, 3.91, 3.85 [19:23:57] Beta at some point next year will be it's own CentralAuth/CreateWiki/ManageWiki cluster [19:24:23] So a tickbox wouldn't make sense and would likely take a wiki down if you moved it between prod and beta without careful Sysadmin work [19:24:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.60, 23.10, 22.17 [19:24:39] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.77, 1.92, 1.86 [19:25:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:25:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.97, 4.37, 4.03 [19:28:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.70, 1.55, 1.70 [19:28:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.75, 5.60, 5.89 [19:29:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.14, 3.60, 3.79 [19:30:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.96, 18.34, 20.38 [19:32:55] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.04, 5.45, 5.71 [19:33:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:34:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.14, 1.95, 1.78 [19:35:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.57, 3.67 [19:36:39] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.00, 1.66, 1.70 [19:37:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.50, 3.62 [19:38:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:38:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.84, 5.54, 5.80 [19:39:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.97, 3.89, 3.74 [19:40:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.32, 6.00, 5.92 [19:41:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:41:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.54, 3.28, 3.54 [19:42:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.97, 5.74, 5.85 [19:44:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.95, 22.40, 21.12 [19:46:07] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 17.50, 10.47, 7.57 [19:46:36] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 21.35, 12.80, 8.27 [19:46:39] PROBLEM - mw10 APT on mw10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:47:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.47, 3.56, 3.53 [19:48:26] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JDFNh [19:48:28] [02miraheze/CreateWiki] 07Universal-Omega 030b17ae2 - Add auto approval description blacklist to AI [19:48:29] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vpJTL [19:48:31] [02CreateWiki] 07Universal-Omega opened pull request 03#269: Add auto approval description blacklist to AI - 13https://git.io/JDFNj [19:48:42] RECOVERY - mw10 APT on mw10 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [19:48:55] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.29, 6.42, 6.05 [19:49:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.93, 3.30, 3.44 [19:50:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.13, 5.67, 5.83 [19:51:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.86, 4.52, 3.88 [19:52:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.26, 19.25, 20.07 [19:54:03] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.22, 6.82, 7.17 [19:54:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 3.96, 7.30, 7.56 [19:55:02] [02dns] 07Reception123 closed pull request 03#237: Remove expired oecumene.org - 13https://git.io/JDHTe [19:55:03] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/JDFxu [19:55:05] [02miraheze/dns] 07Universal-Omega 030f6c7e3 - Remove expired oecumene.org (#237) [19:57:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.82, 3.78 [19:58:00] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.24, 6.85, 7.06 [19:59:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.97, 3.83 [19:59:59] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.04, 6.30, 6.82 [20:01:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:01:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.82, 3.81 [20:01:58] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.65, 6.21, 6.71 [20:02:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.38, 6.10, 6.78 [20:02:55] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.32, 5.30, 5.47 [20:03:05] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JDFpo [20:03:06] [02miraheze/CreateWiki] 07Universal-Omega 036dacf08 - Add realm option to setVariables to set SiteConfiguration tags [20:03:08] [02CreateWiki] 07Universal-Omega synchronize pull request 03#268: [DNM] Add betaheze to SiteConfiguration - 13https://git.io/JDFSi [20:03:35] [02CreateWiki] 07Universal-Omega edited pull request 03#268: Add realm option to setVariables to set SiteConfiguration tags - 13https://git.io/JDFSi [20:03:45] [02CreateWiki] 07Universal-Omega edited pull request 03#268: Add realm option to setVariables to set SiteConfiguration tags - 13https://git.io/JDFSi [20:04:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:04:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.56, 20.53, 20.14 [20:04:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.78, 5.17, 5.41 [20:05:25] RhinosF1, JohnLewis: ^ should support configuration of it from LS via $wi->setVariables, another parameter, setting [ 'beta' => 'betaheze.org' ] [20:06:55] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.48, 5.67, 5.56 [20:08:32] miraheze/CreateWiki - Universal-Omega the build passed. [20:08:52] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.49, 7.04, 6.94 [20:08:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.36, 5.60, 5.55 [20:10:45] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JDFhb [20:10:47] [02miraheze/CreateWiki] 07Universal-Omega 03d11bc86 - Update extension.json [20:10:48] [02CreateWiki] 07Universal-Omega synchronize pull request 03#269: Add auto approval description blacklist to AI - 13https://git.io/JDFNj [20:10:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.56, 5.88, 5.65 [20:11:21] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.40, 7.20, 6.39 [20:12:49] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.55, 7.86, 7.26 [20:13:19] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.45, 8.70, 7.04 [20:14:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.14, 22.36, 21.20 [20:14:46] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.89, 8.11, 6.78 [20:16:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.00, 22.09, 21.23 [20:16:38] miraheze/CreateWiki - Universal-Omega the build passed. [20:16:49] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JDFjj [20:16:51] [02miraheze/CreateWiki] 07Universal-Omega 03235906a - Update WikiInitialise.php [20:16:52] [02CreateWiki] 07Universal-Omega synchronize pull request 03#268: Add realm option to setVariables to set SiteConfiguration tags - 13https://git.io/JDFSi [20:16:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.60, 5.53, 5.60 [20:17:05] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.54, 7.07, 6.16 [20:17:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.32, 2.98, 3.29 [20:18:43] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.92, 7.90, 7.00 [20:18:50] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JDbeO [20:18:52] [02miraheze/mw-config] 07Universal-Omega 03787e070 - Configue beta realm for SiteConfiguration tag [20:18:53] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [20:18:55] [02mw-config] 07Universal-Omega opened pull request 03#4293: Configue beta realm for SiteConfiguration tag - 13https://git.io/JDbe3 [20:19:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:19:03] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.03, 7.51, 6.42 [20:19:48] [02mw-config] 07Universal-Omega edited pull request 03#4293: Configue beta realm for SiteConfiguration tag - 13https://git.io/JDbe3 [20:20:11] miraheze/mw-config - Universal-Omega the build passed. [20:20:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.89, 5.71, 5.61 [20:21:02] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.33, 7.25, 6.44 [20:21:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:21:53] miraheze/CreateWiki - Universal-Omega the build passed. [20:22:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.06, 5.02, 5.37 [20:23:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.82, 4.00, 3.63 [20:24:03] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.72, 7.25, 6.67 [20:24:54] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.99, 7.43, 6.27 [20:24:54] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.63, 4.39, 5.09 [20:25:00] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.92, 7.70, 6.73 [20:26:52] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.08, 7.32, 6.37 [20:26:59] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.13, 7.00, 6.59 [20:27:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.66, 7.68, 7.75 [20:27:58] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.65, 6.45, 6.49 [20:28:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.76, 5.90, 5.50 [20:28:57] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.73, 6.51, 6.45 [20:29:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.87, 3.69, 3.68 [20:30:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.60, 5.32, 5.34 [20:32:47] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.33, 6.61, 6.41 [20:32:55] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.85, 4.56, 5.06 [20:34:33] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.46, 7.26, 7.04 [20:36:32] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.08, 7.73, 7.97 [20:36:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.26, 19.00, 20.22 [20:36:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.17, 7.33, 7.48 [20:38:31] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.46, 8.43, 8.20 [20:38:31] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.06, 7.54, 7.24 [20:38:46] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.46, 7.22, 7.41 [20:42:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.10, 7.92, 7.42 [20:42:28] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.35, 7.56, 7.91 [20:43:01] CosmicAlpha: so just do s/betawiki/beta in the config [20:43:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.57, 3.94, 3.77 [20:44:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.83, 7.51, 7.33 [20:44:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.83, 19.98, 20.07 [20:46:25] RhinosF1: https://github.com/miraheze/mw-config/pull/4293, in conjunction with the CW PR should support `beta` tag for those on betaheze.org. If I did that right anyway. [20:46:26] [url] Configue beta realm for SiteConfiguration tag by Universal-Omega · Pull Request #4293 · miraheze/mw-config · GitHub | github.com [20:46:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.18, 19.27, 19.81 [20:48:24] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.28, 7.31, 7.57 [20:48:47] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.18, 7.03, 6.78 [20:49:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.87, 3.87 [20:50:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.50, 7.39, 7.23 [20:50:30] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.21, 8.16, 7.65 [20:50:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.41, 7.09, 6.70 [20:50:41] CosmicAlpha: cool :) [20:50:48] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.50, 6.61, 6.67 [20:51:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:51:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.89, 4.02, 3.93 [20:52:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:52:21] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.93, 7.48, 7.28 [20:52:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.20, 21.69, 20.60 [20:54:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.22, 7.46, 6.93 [20:55:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.95, 3.93 [20:56:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.71, 7.62, 7.06 [20:57:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:57:43] [02mw-config] 07Universal-Omega edited pull request 03#4293: Configure beta realm for SiteConfiguration tag - 13https://git.io/JDbe3 [20:57:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.91, 4.20, 4.02 [20:58:34] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 14.13, 9.38, 7.72 [20:59:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:00:21] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.87, 7.19, 7.59 [21:02:17] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.91, 5.77, 6.59 [21:03:40] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.98, 7.97, 7.16 [21:07:37] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.20, 7.52, 7.17 [21:07:46] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.49, 6.56, 5.38 [21:08:13] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.72, 6.07, 7.78 [21:09:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:09:40] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.09, 5.62, 5.18 [21:11:35] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.12, 5.84, 5.28 [21:13:29] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.91, 5.35, 5.16 [21:14:21] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.07, 7.01, 7.04 [21:16:21] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.07, 6.88, 7.01 [21:16:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.80, 19.53, 20.36 [21:17:30] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.22, 6.09, 6.68 [21:18:21] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.71, 6.24, 6.76 [21:19:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:20:04] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.02, 6.58, 7.08 [21:20:34] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.12, 6.59, 7.71 [21:21:07] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.32, 4.78, 4.99 [21:22:03] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.22, 5.96, 6.80 [21:24:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:26:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.15, 5.48, 6.80 [21:43:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:48:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:49:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:49:39] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.11, 7.33, 6.72 [21:49:44] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.94, 6.69, 6.34 [21:50:40] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.53, 23.12, 20.59 [21:51:38] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.19, 7.09, 6.72 [21:52:36] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.31, 22.45, 20.68 [21:53:37] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.36, 6.34, 6.48 [21:53:41] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.71, 6.17, 6.25 [21:53:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.65, 3.72, 4.00 [21:54:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:56:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.98, 20.05, 20.08 [21:59:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.28, 3.75, 3.88 [22:01:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 4.00, 3.97 [22:05:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.11, 4.22, 4.04 [22:09:24] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.33, 6.87, 6.39 [22:11:23] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.85, 6.92, 6.45 [22:13:21] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.88, 6.58, 6.39 [22:13:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.04, 3.58, 3.86 [22:23:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.60, 2.69, 3.36 [22:25:10] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.56, 4.84, 4.43 [22:27:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:28:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.60, 21.72, 19.44 [22:30:21] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.36, 6.75, 6.07 [22:30:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.50, 21.47, 19.61 [22:30:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.32, 6.03, 5.09 [22:32:02] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:32:33] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.34, 19.97, 19.29 [22:32:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.87, 5.64, 5.07 [22:34:30] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.67, 3.54 [22:34:55] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.94, 6.27, 5.37 [22:36:22] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.06, 6.68, 6.29 [22:36:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.06, 5.84, 5.33 [22:38:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.73, 3.61 [22:40:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.75, 3.62 [22:42:17] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.88, 6.95, 6.31 [22:42:43] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:42:55] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.62, 7.14, 5.89 [22:43:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:44:01] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 5.69, 4.10, 2.47 [22:44:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.75, 3.47, 3.55 [22:44:17] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.03, 6.36, 6.16 [22:45:55] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 2.74, 3.57, 2.47 [22:47:49] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 3.08, 3.37, 2.52 [22:51:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.06, 2.95, 3.30 [22:53:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:54:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:00:04] !log [@mw11] starting deploy of {'l10nupdate': True} to ovlon [23:00:05] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:12] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.71, 3.76, 2.98 [23:03:09] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:10] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:24] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:03:44] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:47] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.73, 7.85, 6.35 [23:04:39] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:05:17] PROBLEM - test3 APT on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:05:44] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 12.02, 8.32, 6.16 [23:06:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.77, 7.62, 6.41 [23:07:12] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 2.99, 3.62, 3.24 [23:07:15] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 4.308 second response time [23:07:16] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 20514 bytes in 2.078 second response time [23:07:58] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 4.905 second response time [23:08:18] PROBLEM - test3 Current Load on test3 is CRITICAL: CRITICAL - load average: 7.28, 3.83, 1.63 [23:08:33] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.03, 19.72, 16.16 [23:09:12] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 1.97, 3.14, 3.11 [23:09:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.12, 4.16, 3.43 [23:10:04] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:10:06] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:10:30] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.19, 21.65, 17.28 [23:11:35] RECOVERY - test3 APT on test3 is OK: APT OK: 19 packages available for upgrade (0 critical updates). [23:12:05] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.79, 8.94, 6.85 [23:12:17] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.59, 6.70, 6.09 [23:12:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:12:26] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.48, 21.19, 17.66 [23:13:14] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.69, 7.04, 5.65 [23:13:15] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:13:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.91, 3.77, 3.47 [23:13:51] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:02] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:08] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:14:17] RECOVERY - test3 Current Load on test3 is OK: OK - load average: 0.58, 3.27, 2.34 [23:15:12] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.61, 7.17, 5.86 [23:15:49] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.307 second response time [23:15:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.79, 3.18, 3.29 [23:16:02] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.15, 7.76, 6.86 [23:16:03] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.907 second response time [23:16:05] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 2.403 second response time [23:16:19] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.86, 20.24, 18.08 [23:16:24] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.883 second response time [23:16:25] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.148 second response time [23:17:11] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.41, 7.33, 6.06 [23:18:17] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.00, 7.76, 6.73 [23:19:09] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 8.00, 7.61, 6.32 [23:20:17] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.62, 7.59, 6.79 [23:21:58] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.69, 7.88, 7.16 [23:23:06] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::5ebc/cpweb [23:23:08] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb [23:23:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.25, 3.89, 3.52 [23:23:57] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.74, 7.58, 7.12 [23:25:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.36, 7.39, 6.57 [23:25:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:26:20] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:27:03] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:27:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.73, 3.54 [23:27:55] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.36, 8.22, 7.45 [23:28:03] PROBLEM - test3 APT on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:28:12] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:28:17] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.61, 6.04, 6.44 [23:28:28] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:28:57] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.10, 22.23, 19.72 [23:29:29] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.84, 6.68, 7.64 [23:29:44] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:29:53] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.00, 7.36, 7.23 [23:29:56] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:30:18] PROBLEM - test3 Current Load on test3 is CRITICAL: CRITICAL - load average: 10.14, 5.70, 3.22 [23:31:44] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 5.933 second response time [23:31:52] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.20, 8.79, 7.79 [23:31:57] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.662 second response time [23:32:04] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.498 second response time [23:32:13] RECOVERY - test3 APT on test3 is OK: APT OK: 19 packages available for upgrade (0 critical updates). [23:32:36] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.079 second response time [23:32:44] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.519 second response time [23:32:50] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.51, 22.98, 20.67 [23:32:58] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.33, 7.72, 7.14 [23:33:51] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.82, 7.82, 7.54 [23:34:21] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.02, 6.71, 7.93 [23:35:09] !log [@mw11] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 2105s [23:35:10] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 2105s [23:35:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:35:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:35:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.42, 3.12, 3.37 [23:36:03] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:36:17] PROBLEM - test3 Current Load on test3 is WARNING: WARNING - load average: 0.52, 3.54, 3.32 [23:36:21] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:36:22] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:37:24] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.95, 5.58, 6.73 [23:37:58] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.684 second response time [23:38:17] RECOVERY - test3 Current Load on test3 is OK: OK - load average: 0.07, 2.36, 2.91 [23:38:18] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.199 second response time [23:38:20] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.356 second response time [23:38:21] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.13, 7.59, 7.95 [23:40:52] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.70, 6.41, 6.72 [23:41:20] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.80, 6.58, 6.95 [23:42:21] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.11, 7.35, 7.80 [23:43:19] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.67, 6.28, 6.79 [23:44:22] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.03, 7.80, 7.91 [23:44:52] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.30, 6.64, 7.89 [23:46:21] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.51, 7.19, 7.69 [23:46:24] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.22, 19.17, 20.35 [23:49:40] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.05, 5.64, 6.54 [23:50:12] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.31, 1.48, 1.09 [23:51:28] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [23:51:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb [23:52:11] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.72, 1.17, 1.02 [23:52:21] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.75, 5.35, 6.74 [23:52:41] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.85, 5.25, 6.72 [23:53:24] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:53:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:54:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:56:21] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.81, 6.54, 6.88 [23:57:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:58:04] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.29, 1.74, 1.36 [23:58:23] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.81, 6.07, 6.65