[00:00:29] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [00:00:29] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404013395309448 seconds [00:02:36] RECOVERY - matomo101 Disk Space on matomo101 is OK: DISK OK - free space: / 2065 MB (21% inode=80%); [00:04:50] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 495 failures. Last run 3 minutes ago with 495 failures. Failed resources (up to 3 shown): File[/etc/powerdns/recursor.conf],File[/usr/local/bin/gen_fingerprints],File[/usr/local/bin/logsalmsg],File[/etc/vim/vimrc.local] [00:05:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.73, 3.22 [00:06:05] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.36, 3.63, 3.18 [00:07:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.41, 3.68, 3.27 [00:08:04] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.26, 3.43, 3.16 [00:09:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.99, 4.25, 3.53 [00:11:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.30, 3.87, 3.25 [00:14:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.30, 4.27, 3.58 [00:15:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.68, 3.71, 3.36 [00:16:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.98, 3.56, 3.42 [00:17:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.72, 3.26, 3.23 [00:17:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.90, 3.97, 3.77 [00:18:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.07, 3.72, 3.48 [00:20:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.66, 3.52, 3.43 [00:21:23] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.02, 8.56, 8.32 [00:22:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.04, 3.03, 3.26 [00:25:21] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.76, 8.45, 8.35 [00:25:44] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.73, 2.79, 3.31 [00:28:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MatomoAnalytics-880ae20 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9c0533efd691 [00:28:04] [02miraheze/mediawiki] 07dependabot[bot] 039c0533e - Bump extensions/MatomoAnalytics from `f2a07d0` to `880ae20` [00:28:06] [02mediawiki] 07dependabot[bot] opened pull request 03#5539: Bump extensions/MatomoAnalytics from `f2a07d0` to `880ae20` - 13https://github.com/miraheze/mediawiki/pull/5539 [00:28:07] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MatomoAnalytics-880ae20 - 13https://github.com/miraheze/mediawiki [00:28:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ConfirmEdit-7848c32 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8b0506b4d5d4 [00:28:10] [02miraheze/mediawiki] 07dependabot[bot] 038b0506b - Bump extensions/ConfirmEdit from `c0fc66e` to `7848c32` [00:28:12] [02mediawiki] 07dependabot[bot] opened pull request 03#5540: Bump extensions/ConfirmEdit from `c0fc66e` to `7848c32` - 13https://github.com/miraheze/mediawiki/pull/5540 [00:28:13] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ConfirmEdit-7848c32 - 13https://github.com/miraheze/mediawiki [00:28:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Medik-677ebb9 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4f0a0617f1af [00:28:16] [02miraheze/mediawiki] 07dependabot[bot] 034f0a061 - Bump skins/Medik from `3a10ae1` to `677ebb9` [00:28:18] [02mediawiki] 07dependabot[bot] opened pull request 03#5541: Bump skins/Medik from `3a10ae1` to `677ebb9` - 13https://github.com/miraheze/mediawiki/pull/5541 [00:28:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Medik-677ebb9 - 13https://github.com/miraheze/mediawiki [00:28:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikiForum-d525280 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a6ecf546ef82 [00:28:22] [02miraheze/mediawiki] 07dependabot[bot] 03a6ecf54 - Bump extensions/WikiForum from `2a42c61` to `d525280` [00:28:24] [02mediawiki] 07dependabot[bot] opened pull request 03#5542: Bump extensions/WikiForum from `2a42c61` to `d525280` - 13https://github.com/miraheze/mediawiki/pull/5542 [00:28:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikiForum-d525280 - 13https://github.com/miraheze/mediawiki [00:28:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DiscussionTools-4c0f69d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a9fe961d87bc [00:28:28] [02miraheze/mediawiki] 07dependabot[bot] 03a9fe961 - Bump extensions/DiscussionTools from `d57a7e0` to `4c0f69d` [00:28:30] [02mediawiki] 07dependabot[bot] opened pull request 03#5543: Bump extensions/DiscussionTools from `d57a7e0` to `4c0f69d` - 13https://github.com/miraheze/mediawiki/pull/5543 [00:28:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DiscussionTools-4c0f69d - 13https://github.com/miraheze/mediawiki [00:28:33] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ArticleToCategory2-5bc093f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b723ff2e74de [00:28:34] [02miraheze/mediawiki] 07dependabot[bot] 03b723ff2 - Bump extensions/ArticleToCategory2 from `683d878` to `5bc093f` [00:28:36] [02mediawiki] 07dependabot[bot] opened pull request 03#5544: Bump extensions/ArticleToCategory2 from `683d878` to `5bc093f` - 13https://github.com/miraheze/mediawiki/pull/5544 [00:28:37] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ArticleToCategory2-5bc093f - 13https://github.com/miraheze/mediawiki [00:28:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TwoColConflict-6e7687a [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e6f000e1c77f [00:28:40] [02miraheze/mediawiki] 07dependabot[bot] 03e6f000e - Bump extensions/TwoColConflict from `d57f351` to `6e7687a` [00:28:42] [02mediawiki] 07dependabot[bot] opened pull request 03#5545: Bump extensions/TwoColConflict from `d57f351` to `6e7687a` - 13https://github.com/miraheze/mediawiki/pull/5545 [00:28:43] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TwoColConflict-6e7687a - 13https://github.com/miraheze/mediawiki [00:28:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Report-b10a2dc [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/3b5cbd74ff66 [00:28:46] [02miraheze/mediawiki] 07dependabot[bot] 033b5cbd7 - Bump extensions/Report from `9369e49` to `b10a2dc` [00:28:48] [02mediawiki] 07dependabot[bot] opened pull request 03#5546: Bump extensions/Report from `9369e49` to `b10a2dc` - 13https://github.com/miraheze/mediawiki/pull/5546 [00:28:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Report-b10a2dc - 13https://github.com/miraheze/mediawiki [00:28:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/HAWelcome-cc33794 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f206ade4fb6e [00:28:52] [02miraheze/mediawiki] 07dependabot[bot] 03f206ade - Bump extensions/HAWelcome from `67c1410` to `cc33794` [00:28:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#5539: Bump extensions/MatomoAnalytics from `f2a07d0` to `880ae20` - 13https://github.com/miraheze/mediawiki/pull/5539 [00:28:55] [02mediawiki] 07dependabot[bot] opened pull request 03#5547: Bump extensions/HAWelcome from `67c1410` to `cc33794` - 13https://github.com/miraheze/mediawiki/pull/5547 [00:28:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#5539: Bump extensions/MatomoAnalytics from `f2a07d0` to `880ae20` - 13https://github.com/miraheze/mediawiki/pull/5539 [00:28:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/HAWelcome-cc33794 - 13https://github.com/miraheze/mediawiki [00:29:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TemplateSandbox-937f237 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1abb2c192a91 [00:29:01] [02miraheze/mediawiki] 07dependabot[bot] 031abb2c1 - Bump extensions/TemplateSandbox from `16beb2e` to `937f237` [00:29:03] [02mediawiki] 07dependabot[bot] opened pull request 03#5548: Bump extensions/TemplateSandbox from `16beb2e` to `937f237` - 13https://github.com/miraheze/mediawiki/pull/5548 [00:29:04] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TemplateSandbox-937f237 - 13https://github.com/miraheze/mediawiki [00:29:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalUsage-ca3c7d0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ef8fd879f5d0 [00:29:07] [02miraheze/mediawiki] 07dependabot[bot] 03ef8fd87 - Bump extensions/GlobalUsage from `c5c342a` to `ca3c7d0` [00:29:09] [02mediawiki] 07dependabot[bot] opened pull request 03#5549: Bump extensions/GlobalUsage from `c5c342a` to `ca3c7d0` - 13https://github.com/miraheze/mediawiki/pull/5549 [00:29:10] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalUsage-ca3c7d0 - 13https://github.com/miraheze/mediawiki [00:29:12] [02mediawiki] 07github-actions[bot] labeled pull request 03#5540: Bump extensions/ConfirmEdit from `c0fc66e` to `7848c32` - 13https://github.com/miraheze/mediawiki/pull/5540 [00:29:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#5542: Bump extensions/WikiForum from `2a42c61` to `d525280` - 13https://github.com/miraheze/mediawiki/pull/5542 [00:29:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Thanks-841e1f7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9f22e474aaaf [00:29:16] [02miraheze/mediawiki] 07dependabot[bot] 039f22e47 - Bump extensions/Thanks from `bcedec9` to `841e1f7` [00:29:18] [02mediawiki] 07dependabot[bot] opened pull request 03#5550: Bump extensions/Thanks from `bcedec9` to `841e1f7` - 13https://github.com/miraheze/mediawiki/pull/5550 [00:29:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.17, 8.72, 8.46 [00:29:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Thanks-841e1f7 - 13https://github.com/miraheze/mediawiki [00:29:21] [02mediawiki] 07github-actions[bot] labeled pull request 03#5541: Bump skins/Medik from `3a10ae1` to `677ebb9` - 13https://github.com/miraheze/mediawiki/pull/5541 [00:29:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#5543: Bump extensions/DiscussionTools from `d57a7e0` to `4c0f69d` - 13https://github.com/miraheze/mediawiki/pull/5543 [00:29:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-32ee5f5 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1469dbbf0365 [00:29:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.28, 8.06, 7.38 [00:29:25] [02miraheze/mediawiki] 07dependabot[bot] 031469dbb - Bump extensions/UniversalLanguageSelector from `0bd9a68` to `32ee5f5` [00:29:27] [02mediawiki] 07dependabot[bot] opened pull request 03#5551: Bump extensions/UniversalLanguageSelector from `0bd9a68` to `32ee5f5` - 13https://github.com/miraheze/mediawiki/pull/5551 [00:29:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-32ee5f5 - 13https://github.com/miraheze/mediawiki [00:29:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#5545: Bump extensions/TwoColConflict from `d57f351` to `6e7687a` - 13https://github.com/miraheze/mediawiki/pull/5545 [00:29:31] [02mediawiki] 07github-actions[bot] labeled pull request 03#5544: Bump extensions/ArticleToCategory2 from `683d878` to `5bc093f` - 13https://github.com/miraheze/mediawiki/pull/5544 [00:29:33] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikibaseLocalMedia-b11c828 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a046365581da [00:29:34] [02miraheze/mediawiki] 07dependabot[bot] 03a046365 - Bump extensions/WikibaseLocalMedia from `c925f24` to `b11c828` [00:29:36] [02mediawiki] 07dependabot[bot] opened pull request 03#5552: Bump extensions/WikibaseLocalMedia from `c925f24` to `b11c828` - 13https://github.com/miraheze/mediawiki/pull/5552 [00:29:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#5547: Bump extensions/HAWelcome from `67c1410` to `cc33794` - 13https://github.com/miraheze/mediawiki/pull/5547 [00:29:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikibaseLocalMedia-b11c828 - 13https://github.com/miraheze/mediawiki [00:29:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikiDiscover-87282ea [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/06e1bbfcce87 [00:29:42] [02miraheze/mediawiki] 07dependabot[bot] 0306e1bbf - Bump extensions/WikiDiscover from `086af25` to `87282ea` [00:29:43] [02mediawiki] 07dependabot[bot] opened pull request 03#5553: Bump extensions/WikiDiscover from `086af25` to `87282ea` - 13https://github.com/miraheze/mediawiki/pull/5553 [00:29:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikiDiscover-87282ea - 13https://github.com/miraheze/mediawiki [00:29:46] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Editcount-bb655ff [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d3dafa508d94 [00:29:48] [02miraheze/mediawiki] 07dependabot[bot] 03d3dafa5 - Bump extensions/Editcount from `333340b` to `bb655ff` [00:29:49] [02mediawiki] 07dependabot[bot] opened pull request 03#5554: Bump extensions/Editcount from `333340b` to `bb655ff` - 13https://github.com/miraheze/mediawiki/pull/5554 [00:29:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#5546: Bump extensions/Report from `9369e49` to `b10a2dc` - 13https://github.com/miraheze/mediawiki/pull/5546 [00:29:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Editcount-bb655ff - 13https://github.com/miraheze/mediawiki [00:29:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/UserMerge-15691ba [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e90561b68a17 [00:29:55] [02miraheze/mediawiki] 07dependabot[bot] 03e90561b - Bump extensions/UserMerge from `614e434` to `15691ba` [00:29:57] [02mediawiki] 07dependabot[bot] opened pull request 03#5555: Bump extensions/UserMerge from `614e434` to `15691ba` - 13https://github.com/miraheze/mediawiki/pull/5555 [00:29:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/UserMerge-15691ba - 13https://github.com/miraheze/mediawiki [00:30:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#5549: Bump extensions/GlobalUsage from `c5c342a` to `ca3c7d0` - 13https://github.com/miraheze/mediawiki/pull/5549 [00:30:01] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/VisualEditor-40ee3ad [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/678748e18e28 [00:30:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.89, 3.45, 3.19 [00:30:03] [02miraheze/mediawiki] 07dependabot[bot] 03678748e - Bump extensions/VisualEditor from `04404e3` to `40ee3ad` [00:30:04] [02mediawiki] 07dependabot[bot] opened pull request 03#5556: Bump extensions/VisualEditor from `04404e3` to `40ee3ad` - 13https://github.com/miraheze/mediawiki/pull/5556 [00:30:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/VisualEditor-40ee3ad - 13https://github.com/miraheze/mediawiki [00:30:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#5550: Bump extensions/Thanks from `bcedec9` to `841e1f7` - 13https://github.com/miraheze/mediawiki/pull/5550 [00:30:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Cosmos-182cf3d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9fa980779ba7 [00:30:10] [02miraheze/mediawiki] 07dependabot[bot] 039fa9807 - Bump skins/Cosmos from `34f65a4` to `182cf3d` [00:30:12] [02mediawiki] 07dependabot[bot] opened pull request 03#5557: Bump skins/Cosmos from `34f65a4` to `182cf3d` - 13https://github.com/miraheze/mediawiki/pull/5557 [00:30:13] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Cosmos-182cf3d - 13https://github.com/miraheze/mediawiki [00:30:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageSchemas-bda97b1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c9a22e898b9a [00:30:16] [02miraheze/mediawiki] 07dependabot[bot] 03c9a22e8 - Bump extensions/PageSchemas from `f681392` to `bda97b1` [00:30:18] [02mediawiki] 07dependabot[bot] opened pull request 03#5558: Bump extensions/PageSchemas from `f681392` to `bda97b1` - 13https://github.com/miraheze/mediawiki/pull/5558 [00:30:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageSchemas-bda97b1 - 13https://github.com/miraheze/mediawiki [00:30:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikibaseQualityConstraints-f25e5c1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a5a42c36ecd2 [00:30:22] [02miraheze/mediawiki] 07dependabot[bot] 03a5a42c3 - Bump extensions/WikibaseQualityConstraints from `4579c43` to `f25e5c1` [00:30:24] [02mediawiki] 07dependabot[bot] opened pull request 03#5559: Bump extensions/WikibaseQualityConstraints from `4579c43` to `f25e5c1` - 13https://github.com/miraheze/mediawiki/pull/5559 [00:30:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikibaseQualityConstraints-f25e5c1 - 13https://github.com/miraheze/mediawiki [00:30:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ArticlePlaceholder-f17427e [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4d237c838890 [00:30:28] [02miraheze/mediawiki] 07dependabot[bot] 034d237c8 - Bump extensions/ArticlePlaceholder from `b578850` to `f17427e` [00:30:30] [02mediawiki] 07dependabot[bot] opened pull request 03#5560: Bump extensions/ArticlePlaceholder from `b578850` to `f17427e` - 13https://github.com/miraheze/mediawiki/pull/5560 [00:30:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ArticlePlaceholder-f17427e - 13https://github.com/miraheze/mediawiki [00:30:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#5552: Bump extensions/WikibaseLocalMedia from `c925f24` to `b11c828` - 13https://github.com/miraheze/mediawiki/pull/5552 [00:30:34] [02mediawiki] 07dependabot[bot] opened pull request 03#5561: Bump extensions/ContactPage from `ddff49e` to `6f14242` - 13https://github.com/miraheze/mediawiki/pull/5561 [00:30:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ContactPage-6f14242 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e6856f57b697 [00:30:37] [02miraheze/mediawiki] 07dependabot[bot] 03e6856f5 - Bump extensions/ContactPage from `ddff49e` to `6f14242` [00:30:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ContactPage-6f14242 - 13https://github.com/miraheze/mediawiki [00:30:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Translate-411d06c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f576f42e45cf [00:30:42] [02miraheze/mediawiki] 07dependabot[bot] 03f576f42 - Bump extensions/Translate from `414f3cb` to `411d06c` [00:30:43] [02mediawiki] 07dependabot[bot] opened pull request 03#5562: Bump extensions/Translate from `414f3cb` to `411d06c` - 13https://github.com/miraheze/mediawiki/pull/5562 [00:30:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Translate-411d06c - 13https://github.com/miraheze/mediawiki [00:30:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#5553: Bump extensions/WikiDiscover from `086af25` to `87282ea` - 13https://github.com/miraheze/mediawiki/pull/5553 [00:30:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#5555: Bump extensions/UserMerge from `614e434` to `15691ba` - 13https://github.com/miraheze/mediawiki/pull/5555 [00:30:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#5554: Bump extensions/Editcount from `333340b` to `bb655ff` - 13https://github.com/miraheze/mediawiki/pull/5554 [00:30:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SandboxLink-cc57c5d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/352c8323ecdf [00:30:52] [02miraheze/mediawiki] 07dependabot[bot] 03352c832 - Bump extensions/SandboxLink from `26d8d66` to `cc57c5d` [00:30:54] [02mediawiki] 07dependabot[bot] opened pull request 03#5563: Bump extensions/SandboxLink from `26d8d66` to `cc57c5d` - 13https://github.com/miraheze/mediawiki/pull/5563 [00:30:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SandboxLink-cc57c5d - 13https://github.com/miraheze/mediawiki [00:30:57] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ProofreadPage-866ad8a [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b2b61cfcf46e [00:30:58] [02miraheze/mediawiki] 07dependabot[bot] 03b2b61cf - Bump extensions/ProofreadPage from `a3c08b1` to `866ad8a` [00:31:00] [02mediawiki] 07dependabot[bot] opened pull request 03#5564: Bump extensions/ProofreadPage from `a3c08b1` to `866ad8a` - 13https://github.com/miraheze/mediawiki/pull/5564 [00:31:01] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ProofreadPage-866ad8a - 13https://github.com/miraheze/mediawiki [00:31:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#5556: Bump extensions/VisualEditor from `04404e3` to `40ee3ad` - 13https://github.com/miraheze/mediawiki/pull/5556 [00:31:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/FlaggedRevs-a45f5fc [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e36b96f283d6 [00:31:06] [02miraheze/mediawiki] 07dependabot[bot] 03e36b96f - Bump extensions/FlaggedRevs from `1220934` to `a45f5fc` [00:31:07] [02mediawiki] 07dependabot[bot] opened pull request 03#5565: Bump extensions/FlaggedRevs from `1220934` to `a45f5fc` - 13https://github.com/miraheze/mediawiki/pull/5565 [00:31:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/FlaggedRevs-a45f5fc - 13https://github.com/miraheze/mediawiki [00:31:10] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Newsletter-31d866c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b09f138b7219 [00:31:12] [02miraheze/mediawiki] 07dependabot[bot] 03b09f138 - Bump extensions/Newsletter from `be1e137` to `31d866c` [00:31:13] [02mediawiki] 07dependabot[bot] opened pull request 03#5566: Bump extensions/Newsletter from `be1e137` to `31d866c` - 13https://github.com/miraheze/mediawiki/pull/5566 [00:31:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Newsletter-31d866c - 13https://github.com/miraheze/mediawiki [00:31:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/RatePage-238fa8d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5f3e90a21e93 [00:31:18] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.48, 9.23, 8.67 [00:31:18] [02miraheze/mediawiki] 07dependabot[bot] 035f3e90a - Bump extensions/RatePage from `5c1e4b0` to `238fa8d` [00:31:19] [02mediawiki] 07dependabot[bot] opened pull request 03#5567: Bump extensions/RatePage from `5c1e4b0` to `238fa8d` - 13https://github.com/miraheze/mediawiki/pull/5567 [00:31:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/RatePage-238fa8d - 13https://github.com/miraheze/mediawiki [00:31:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#5557: Bump skins/Cosmos from `34f65a4` to `182cf3d` - 13https://github.com/miraheze/mediawiki/pull/5557 [00:31:24] [02mediawiki] 07github-actions[bot] labeled pull request 03#5557: Bump skins/Cosmos from `34f65a4` to `182cf3d` - 13https://github.com/miraheze/mediawiki/pull/5557 [00:31:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.68, 7.77, 7.35 [00:31:25] [02mediawiki] 07github-actions[bot] labeled pull request 03#5558: Bump extensions/PageSchemas from `f681392` to `bda97b1` - 13https://github.com/miraheze/mediawiki/pull/5558 [00:31:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageForms-1d74db6 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/036f0878986c [00:31:28] [02miraheze/mediawiki] 07dependabot[bot] 03036f087 - Bump extensions/PageForms from `6bd0703` to `1d74db6` [00:31:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#5559: Bump extensions/WikibaseQualityConstraints from `4579c43` to `f25e5c1` - 13https://github.com/miraheze/mediawiki/pull/5559 [00:31:31] [02mediawiki] 07dependabot[bot] opened pull request 03#5568: Bump extensions/PageForms from `6bd0703` to `1d74db6` - 13https://github.com/miraheze/mediawiki/pull/5568 [00:31:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageForms-1d74db6 - 13https://github.com/miraheze/mediawiki [00:31:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#5561: Bump extensions/ContactPage from `ddff49e` to `6f14242` - 13https://github.com/miraheze/mediawiki/pull/5561 [00:31:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GettingStarted-05c50cf [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/18ad52b1b183 [00:31:37] [02miraheze/mediawiki] 07dependabot[bot] 0318ad52b - Bump extensions/GettingStarted from `4ecba48` to `05c50cf` [00:31:39] [02mediawiki] 07dependabot[bot] opened pull request 03#5569: Bump extensions/GettingStarted from `4ecba48` to `05c50cf` - 13https://github.com/miraheze/mediawiki/pull/5569 [00:31:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#5560: Bump extensions/ArticlePlaceholder from `b578850` to `f17427e` - 13https://github.com/miraheze/mediawiki/pull/5560 [00:31:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GettingStarted-05c50cf - 13https://github.com/miraheze/mediawiki [00:31:43] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TranslationNotifications-7956b32 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/14adc13f4de0 [00:31:45] [02miraheze/mediawiki] 07dependabot[bot] 0314adc13 - Bump extensions/TranslationNotifications from `2f58a4d` to `7956b32` [00:31:46] [02mediawiki] 07dependabot[bot] opened pull request 03#5570: Bump extensions/TranslationNotifications from `2f58a4d` to `7956b32` - 13https://github.com/miraheze/mediawiki/pull/5570 [00:31:48] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TranslationNotifications-7956b32 - 13https://github.com/miraheze/mediawiki [00:31:49] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/BlueSky-5cc5f2f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ca73914424f6 [00:31:51] [02miraheze/mediawiki] 07dependabot[bot] 03ca73914 - Bump skins/BlueSky from `7afc98b` to `5cc5f2f` [00:31:52] [02mediawiki] 07dependabot[bot] opened pull request 03#5571: Bump skins/BlueSky from `7afc98b` to `5cc5f2f` - 13https://github.com/miraheze/mediawiki/pull/5571 [00:31:54] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/BlueSky-5cc5f2f - 13https://github.com/miraheze/mediawiki [00:31:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#5563: Bump extensions/SandboxLink from `26d8d66` to `cc57c5d` - 13https://github.com/miraheze/mediawiki/pull/5563 [00:31:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#5566: Bump extensions/Newsletter from `be1e137` to `31d866c` - 13https://github.com/miraheze/mediawiki/pull/5566 [00:31:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/MinervaNeue-e13fe16 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/933b981863dc [00:32:00] [02miraheze/mediawiki] 07dependabot[bot] 03933b981 - Bump skins/MinervaNeue from `060f0b3` to `e13fe16` [00:32:01] [02mediawiki] 07dependabot[bot] opened pull request 03#5572: Bump skins/MinervaNeue from `060f0b3` to `e13fe16` - 13https://github.com/miraheze/mediawiki/pull/5572 [00:32:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.08, 3.32, 3.18 [00:32:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/MinervaNeue-e13fe16 - 13https://github.com/miraheze/mediawiki [00:32:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#5564: Bump extensions/ProofreadPage from `a3c08b1` to `866ad8a` - 13https://github.com/miraheze/mediawiki/pull/5564 [00:32:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Maps-ddd5e63 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/de6309c6eda2 [00:32:07] [02miraheze/mediawiki] 07dependabot[bot] 03de6309c - Bump extensions/Maps from `9eae313` to `ddd5e63` [00:32:09] [02mediawiki] 07dependabot[bot] opened pull request 03#5573: Bump extensions/Maps from `9eae313` to `ddd5e63` - 13https://github.com/miraheze/mediawiki/pull/5573 [00:32:10] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Maps-ddd5e63 - 13https://github.com/miraheze/mediawiki [00:32:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/BetaFeatures-9f1c89f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/cec9b75b0fea [00:32:13] [02miraheze/mediawiki] 07dependabot[bot] 03cec9b75 - Bump extensions/BetaFeatures from `178df60` to `9f1c89f` [00:32:15] [02mediawiki] 07dependabot[bot] opened pull request 03#5574: Bump extensions/BetaFeatures from `178df60` to `9f1c89f` - 13https://github.com/miraheze/mediawiki/pull/5574 [00:32:16] [02mediawiki] 07github-actions[bot] labeled pull request 03#5565: Bump extensions/FlaggedRevs from `1220934` to `a45f5fc` - 13https://github.com/miraheze/mediawiki/pull/5565 [00:32:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/BetaFeatures-9f1c89f - 13https://github.com/miraheze/mediawiki [00:32:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#5562: Bump extensions/Translate from `414f3cb` to `411d06c` - 13https://github.com/miraheze/mediawiki/pull/5562 [00:32:21] [02mediawiki] 07github-actions[bot] labeled pull request 03#5562: Bump extensions/Translate from `414f3cb` to `411d06c` - 13https://github.com/miraheze/mediawiki/pull/5562 [00:32:22] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MassMessage-c70e3be [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9d4078423fc2 [00:32:24] [02miraheze/mediawiki] 07dependabot[bot] 039d40784 - Bump extensions/MassMessage from `b11fa3d` to `c70e3be` [00:32:25] [02mediawiki] 07dependabot[bot] opened pull request 03#5575: Bump extensions/MassMessage from `b11fa3d` to `c70e3be` - 13https://github.com/miraheze/mediawiki/pull/5575 [00:32:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MassMessage-c70e3be - 13https://github.com/miraheze/mediawiki [00:32:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PollNY-a8b1780 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d7882d173193 [00:32:30] [02miraheze/mediawiki] 07dependabot[bot] 03d7882d1 - Bump extensions/PollNY from `3df00e9` to `a8b1780` [00:32:31] [02mediawiki] 07github-actions[bot] labeled pull request 03#5569: Bump extensions/GettingStarted from `4ecba48` to `05c50cf` - 13https://github.com/miraheze/mediawiki/pull/5569 [00:32:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#5567: Bump extensions/RatePage from `5c1e4b0` to `238fa8d` - 13https://github.com/miraheze/mediawiki/pull/5567 [00:32:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#5548: Bump extensions/TemplateSandbox from `16beb2e` to `937f237` - 13https://github.com/miraheze/mediawiki/pull/5548 [00:32:36] [02mediawiki] 07dependabot[bot] opened pull request 03#5576: Bump extensions/PollNY from `3df00e9` to `a8b1780` - 13https://github.com/miraheze/mediawiki/pull/5576 [00:32:37] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PollNY-a8b1780 - 13https://github.com/miraheze/mediawiki [00:32:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#5571: Bump skins/BlueSky from `7afc98b` to `5cc5f2f` - 13https://github.com/miraheze/mediawiki/pull/5571 [00:32:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/LocalisationUpdate-e5da971 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e4b0359398cf [00:32:42] [02miraheze/mediawiki] 07dependabot[bot] 03e4b0359 - Bump extensions/LocalisationUpdate from `bed29e2` to `e5da971` [00:32:43] [02mediawiki] 07dependabot[bot] opened pull request 03#5577: Bump extensions/LocalisationUpdate from `bed29e2` to `e5da971` - 13https://github.com/miraheze/mediawiki/pull/5577 [00:32:44] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:32:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#5568: Bump extensions/PageForms from `6bd0703` to `1d74db6` - 13https://github.com/miraheze/mediawiki/pull/5568 [00:32:46] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/LocalisationUpdate-e5da971 - 13https://github.com/miraheze/mediawiki [00:32:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#5568: Bump extensions/PageForms from `6bd0703` to `1d74db6` - 13https://github.com/miraheze/mediawiki/pull/5568 [00:32:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#5570: Bump extensions/TranslationNotifications from `2f58a4d` to `7956b32` - 13https://github.com/miraheze/mediawiki/pull/5570 [00:32:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalBlocking-e98c6bc [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c164619f0128 [00:32:52] [02miraheze/mediawiki] 07dependabot[bot] 03c164619 - Bump extensions/GlobalBlocking from `51acef6` to `e98c6bc` [00:32:54] [02mediawiki] 07dependabot[bot] opened pull request 03#5578: Bump extensions/GlobalBlocking from `51acef6` to `e98c6bc` - 13https://github.com/miraheze/mediawiki/pull/5578 [00:32:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalBlocking-e98c6bc - 13https://github.com/miraheze/mediawiki [00:32:57] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MintyDocs-22c7541 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/23102ad4fdf2 [00:32:58] [02miraheze/mediawiki] 07dependabot[bot] 0323102ad - Bump extensions/MintyDocs from `af2dff9` to `22c7541` [00:33:00] [02mediawiki] 07dependabot[bot] opened pull request 03#5579: Bump extensions/MintyDocs from `af2dff9` to `22c7541` - 13https://github.com/miraheze/mediawiki/pull/5579 [00:33:01] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MintyDocs-22c7541 - 13https://github.com/miraheze/mediawiki [00:33:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Babel-265a94d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c584922a53cc [00:33:04] [02miraheze/mediawiki] 07dependabot[bot] 03c584922 - Bump extensions/Babel from `ef5276d` to `265a94d` [00:33:06] [02mediawiki] 07dependabot[bot] opened pull request 03#5580: Bump extensions/Babel from `ef5276d` to `265a94d` - 13https://github.com/miraheze/mediawiki/pull/5580 [00:33:07] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Babel-265a94d - 13https://github.com/miraheze/mediawiki [00:33:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CentralNotice-f08dcd0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/da871952cfe9 [00:33:10] [02miraheze/mediawiki] 07dependabot[bot] 03da87195 - Bump extensions/CentralNotice from `3d946a0` to `f08dcd0` [00:33:12] [02mediawiki] 07dependabot[bot] opened pull request 03#5581: Bump extensions/CentralNotice from `3d946a0` to `f08dcd0` - 13https://github.com/miraheze/mediawiki/pull/5581 [00:33:13] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CentralNotice-f08dcd0 - 13https://github.com/miraheze/mediawiki [00:33:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#5573: Bump extensions/Maps from `9eae313` to `ddd5e63` - 13https://github.com/miraheze/mediawiki/pull/5573 [00:33:16] [02mediawiki] 07github-actions[bot] labeled pull request 03#5576: Bump extensions/PollNY from `3df00e9` to `a8b1780` - 13https://github.com/miraheze/mediawiki/pull/5576 [00:33:17] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.46, 9.15, 8.70 [00:33:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MsCatSelect-4478a53 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0987a2a0ca3c [00:33:19] [02miraheze/mediawiki] 07dependabot[bot] 030987a2a - Bump extensions/MsCatSelect from `8759480` to `4478a53` [00:33:21] [02mediawiki] 07dependabot[bot] opened pull request 03#5582: Bump extensions/MsCatSelect from `8759480` to `4478a53` - 13https://github.com/miraheze/mediawiki/pull/5582 [00:33:22] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MsCatSelect-4478a53 - 13https://github.com/miraheze/mediawiki [00:33:24] [02mediawiki] 07github-actions[bot] labeled pull request 03#5572: Bump skins/MinervaNeue from `060f0b3` to `e13fe16` - 13https://github.com/miraheze/mediawiki/pull/5572 [00:33:25] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/StructuredNavigation-167f8e2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2e602ee29117 [00:33:27] [02miraheze/mediawiki] 07dependabot[bot] 032e602ee - Bump extensions/StructuredNavigation from `ad1137a` to `167f8e2` [00:33:28] [02mediawiki] 07dependabot[bot] opened pull request 03#5583: Bump extensions/StructuredNavigation from `ad1137a` to `167f8e2` - 13https://github.com/miraheze/mediawiki/pull/5583 [00:33:30] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/StructuredNavigation-167f8e2 - 13https://github.com/miraheze/mediawiki [00:33:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SyntaxHighlight_GeSHi-c26f9a2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/eef3634b2f89 [00:33:33] [02miraheze/mediawiki] 07dependabot[bot] 03eef3634 - Bump extensions/SyntaxHighlight_GeSHi from `c1fe9cd` to `c26f9a2` [00:33:34] [02mediawiki] 07dependabot[bot] opened pull request 03#5584: Bump extensions/SyntaxHighlight_GeSHi from `c1fe9cd` to `c26f9a2` - 13https://github.com/miraheze/mediawiki/pull/5584 [00:33:36] [02mediawiki] 07github-actions[bot] labeled pull request 03#5575: Bump extensions/MassMessage from `b11fa3d` to `c70e3be` - 13https://github.com/miraheze/mediawiki/pull/5575 [00:33:37] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SyntaxHighlight_GeSHi-c26f9a2 - 13https://github.com/miraheze/mediawiki [00:33:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Mirage-7ce40c1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/165199f346ae [00:33:40] [02miraheze/mediawiki] 07dependabot[bot] 03165199f - Bump skins/Mirage from `4c00cd9` to `7ce40c1` [00:33:42] [02mediawiki] 07dependabot[bot] opened pull request 03#5585: Bump skins/Mirage from `4c00cd9` to `7ce40c1` - 13https://github.com/miraheze/mediawiki/pull/5585 [00:33:43] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Mirage-7ce40c1 - 13https://github.com/miraheze/mediawiki [00:33:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-76c1e10 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/da93054bc766 [00:33:46] [02miraheze/mediawiki] 07dependabot[bot] 03da93054 - Bump extensions/MirahezeMagic from `743af44` to `76c1e10` [00:33:48] [02mediawiki] 07dependabot[bot] opened pull request 03#5586: Bump extensions/MirahezeMagic from `743af44` to `76c1e10` - 13https://github.com/miraheze/mediawiki/pull/5586 [00:33:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-76c1e10 - 13https://github.com/miraheze/mediawiki [00:33:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TemplateWizard-2338ec7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/eed354d75777 [00:33:52] [02miraheze/mediawiki] 07dependabot[bot] 03eed354d - Bump extensions/TemplateWizard from `187552c` to `2338ec7` [00:33:54] [02mediawiki] 07dependabot[bot] opened pull request 03#5587: Bump extensions/TemplateWizard from `187552c` to `2338ec7` - 13https://github.com/miraheze/mediawiki/pull/5587 [00:33:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TemplateWizard-2338ec7 - 13https://github.com/miraheze/mediawiki [00:33:57] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AbuseFilter-53e6c3e [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/545095dda302 [00:33:58] [02miraheze/mediawiki] 07dependabot[bot] 03545095d - Bump extensions/AbuseFilter from `dd4da61` to `53e6c3e` [00:34:00] [02mediawiki] 07dependabot[bot] opened pull request 03#5588: Bump extensions/AbuseFilter from `dd4da61` to `53e6c3e` - 13https://github.com/miraheze/mediawiki/pull/5588 [00:34:01] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AbuseFilter-53e6c3e - 13https://github.com/miraheze/mediawiki [00:34:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#5579: Bump extensions/MintyDocs from `af2dff9` to `22c7541` - 13https://github.com/miraheze/mediawiki/pull/5579 [00:34:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/RevisionSlider-5416310 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4ad12d0dc42b [00:34:06] [02miraheze/mediawiki] 07dependabot[bot] 034ad12d0 - Bump extensions/RevisionSlider from `1ca77d5` to `5416310` [00:34:07] [02mediawiki] 07dependabot[bot] opened pull request 03#5589: Bump extensions/RevisionSlider from `1ca77d5` to `5416310` - 13https://github.com/miraheze/mediawiki/pull/5589 [00:34:09] [02mediawiki] 07github-actions[bot] labeled pull request 03#5580: Bump extensions/Babel from `ef5276d` to `265a94d` - 13https://github.com/miraheze/mediawiki/pull/5580 [00:34:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#5578: Bump extensions/GlobalBlocking from `51acef6` to `e98c6bc` - 13https://github.com/miraheze/mediawiki/pull/5578 [00:34:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/RevisionSlider-5416310 - 13https://github.com/miraheze/mediawiki [00:34:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Wikibase-5c27652 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8299b6bfeb30 [00:34:15] [02miraheze/mediawiki] 07dependabot[bot] 038299b6b - Bump extensions/Wikibase from `c496ca4` to `5c27652` [00:34:16] [02mediawiki] 07dependabot[bot] opened pull request 03#5590: Bump extensions/Wikibase from `c496ca4` to `5c27652` - 13https://github.com/miraheze/mediawiki/pull/5590 [00:34:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Wikibase-5c27652 - 13https://github.com/miraheze/mediawiki [00:34:19] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/UrlShortener-e93b161 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ba096cc64910 [00:34:21] [02miraheze/mediawiki] 07dependabot[bot] 03ba096cc - Bump extensions/UrlShortener from `9e223f6` to `e93b161` [00:34:22] [02mediawiki] 07dependabot[bot] opened pull request 03#5591: Bump extensions/UrlShortener from `9e223f6` to `e93b161` - 13https://github.com/miraheze/mediawiki/pull/5591 [00:34:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/UrlShortener-e93b161 - 13https://github.com/miraheze/mediawiki [00:34:25] [02mediawiki] 07github-actions[bot] labeled pull request 03#5582: Bump extensions/MsCatSelect from `8759480` to `4478a53` - 13https://github.com/miraheze/mediawiki/pull/5582 [00:34:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CreateWiki-931a7c7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/138edb29c31f [00:34:28] [02miraheze/mediawiki] 07dependabot[bot] 03138edb2 - Bump extensions/CreateWiki from `8fa2186` to `931a7c7` [00:34:30] [02mediawiki] 07dependabot[bot] opened pull request 03#5592: Bump extensions/CreateWiki from `8fa2186` to `931a7c7` - 13https://github.com/miraheze/mediawiki/pull/5592 [00:34:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CreateWiki-931a7c7 - 13https://github.com/miraheze/mediawiki [00:34:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#5583: Bump extensions/StructuredNavigation from `ad1137a` to `167f8e2` - 13https://github.com/miraheze/mediawiki/pull/5583 [00:34:34] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CiteThisPage-2972451 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/1bde42827dd2 [00:34:36] [02miraheze/mediawiki] 07dependabot[bot] 031bde428 - Bump extensions/CiteThisPage from `bb4d47b` to `2972451` [00:34:37] [02mediawiki] 07dependabot[bot] commented on pull request 03#5513: Bump extensions/CreateWiki from `8fa2186` to `07f0416` - 13https://github.com/miraheze/mediawiki/pull/5513#issuecomment-1053741350 [00:34:39] [02mediawiki] 07dependabot[bot] opened pull request 03#5593: Bump extensions/CiteThisPage from `bb4d47b` to `2972451` - 13https://github.com/miraheze/mediawiki/pull/5593 [00:34:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CiteThisPage-2972451 - 13https://github.com/miraheze/mediawiki [00:34:42] [02mediawiki] 07dependabot[bot] closed pull request 03#5513: Bump extensions/CreateWiki from `8fa2186` to `07f0416` - 13https://github.com/miraheze/mediawiki/pull/5513 [00:34:43] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/CreateWiki-07f0416 [00:34:45] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/CreateWiki-07f0416 - 13https://github.com/miraheze/mediawiki [00:34:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#5584: Bump extensions/SyntaxHighlight_GeSHi from `c1fe9cd` to `c26f9a2` - 13https://github.com/miraheze/mediawiki/pull/5584 [00:34:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#5587: Bump extensions/TemplateWizard from `187552c` to `2338ec7` - 13https://github.com/miraheze/mediawiki/pull/5587 [00:34:49] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalNewFiles-dee5228 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8e2675c5eb35 [00:34:51] [02miraheze/mediawiki] 07dependabot[bot] 038e2675c - Bump extensions/GlobalNewFiles from `68aca4a` to `dee5228` [00:34:52] [02mediawiki] 07dependabot[bot] opened pull request 03#5594: Bump extensions/GlobalNewFiles from `68aca4a` to `dee5228` - 13https://github.com/miraheze/mediawiki/pull/5594 [00:34:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#5585: Bump skins/Mirage from `4c00cd9` to `7ce40c1` - 13https://github.com/miraheze/mediawiki/pull/5585 [00:34:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalNewFiles-dee5228 - 13https://github.com/miraheze/mediawiki [00:34:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#5586: Bump extensions/MirahezeMagic from `743af44` to `76c1e10` - 13https://github.com/miraheze/mediawiki/pull/5586 [00:34:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Disambiguator-55e0f01 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a16ed6cc69cf [00:35:00] [02miraheze/mediawiki] 07dependabot[bot] 03a16ed6c - Bump extensions/Disambiguator from `a6eeae4` to `55e0f01` [00:35:01] [02mediawiki] 07dependabot[bot] opened pull request 03#5595: Bump extensions/Disambiguator from `a6eeae4` to `55e0f01` - 13https://github.com/miraheze/mediawiki/pull/5595 [00:35:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#5586: Bump extensions/MirahezeMagic from `743af44` to `76c1e10` - 13https://github.com/miraheze/mediawiki/pull/5586 [00:35:04] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Disambiguator-55e0f01 - 13https://github.com/miraheze/mediawiki [00:35:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#5586: Bump extensions/MirahezeMagic from `743af44` to `76c1e10` - 13https://github.com/miraheze/mediawiki/pull/5586 [00:35:07] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/RSS-f0b82fb [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/32a14676c52e [00:35:09] [02miraheze/mediawiki] 07dependabot[bot] 0332a1467 - Bump extensions/RSS from `e76c5a6` to `f0b82fb` [00:35:10] [02mediawiki] 07dependabot[bot] opened pull request 03#5596: Bump extensions/RSS from `e76c5a6` to `f0b82fb` - 13https://github.com/miraheze/mediawiki/pull/5596 [00:35:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/RSS-f0b82fb - 13https://github.com/miraheze/mediawiki [00:35:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#5589: Bump extensions/RevisionSlider from `1ca77d5` to `5416310` - 13https://github.com/miraheze/mediawiki/pull/5589 [00:35:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MultimediaViewer-7d5845f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ff65c0275ab4 [00:35:16] [02miraheze/mediawiki] 07dependabot[bot] 03ff65c02 - Bump extensions/MultimediaViewer from `ea6584d` to `7d5845f` [00:35:18] [02mediawiki] 07dependabot[bot] opened pull request 03#5597: Bump extensions/MultimediaViewer from `ea6584d` to `7d5845f` - 13https://github.com/miraheze/mediawiki/pull/5597 [00:35:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MultimediaViewer-7d5845f - 13https://github.com/miraheze/mediawiki [00:35:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CountDownClock-8674d6c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/376b03c5dea2 [00:35:22] [02miraheze/mediawiki] 07dependabot[bot] 03376b03c - Bump extensions/CountDownClock from `451d486` to `8674d6c` [00:35:24] [02mediawiki] 07dependabot[bot] opened pull request 03#5598: Bump extensions/CountDownClock from `451d486` to `8674d6c` - 13https://github.com/miraheze/mediawiki/pull/5598 [00:35:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.95, 8.49, 7.73 [00:35:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CountDownClock-8674d6c - 13https://github.com/miraheze/mediawiki [00:35:27] [02mediawiki] 07github-actions[bot] labeled pull request 03#5591: Bump extensions/UrlShortener from `9e223f6` to `e93b161` - 13https://github.com/miraheze/mediawiki/pull/5591 [00:35:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MobileFrontend-6222894 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/13efd8cb8c70 [00:35:30] [02miraheze/mediawiki] 07dependabot[bot] 0313efd8c - Bump extensions/MobileFrontend from `5d3c0b8` to `6222894` [00:35:31] [02mediawiki] 07dependabot[bot] opened pull request 03#5599: Bump extensions/MobileFrontend from `5d3c0b8` to `6222894` - 13https://github.com/miraheze/mediawiki/pull/5599 [00:35:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MobileFrontend-6222894 - 13https://github.com/miraheze/mediawiki [00:35:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#5588: Bump extensions/AbuseFilter from `dd4da61` to `53e6c3e` - 13https://github.com/miraheze/mediawiki/pull/5588 [00:35:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Linter-5b70116 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c54cb2055241 [00:35:37] [02miraheze/mediawiki] 07dependabot[bot] 03c54cb20 - Bump extensions/Linter from `f0c9993` to `5b70116` [00:35:39] [02mediawiki] 07dependabot[bot] opened pull request 03#5600: Bump extensions/Linter from `f0c9993` to `5b70116` - 13https://github.com/miraheze/mediawiki/pull/5600 [00:35:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Linter-5b70116 - 13https://github.com/miraheze/mediawiki [00:35:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalPreferences-08a0438 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/28d2f16e76f6 [00:35:43] [02miraheze/mediawiki] 07dependabot[bot] 0328d2f16 - Bump extensions/GlobalPreferences from `2503bb0` to `08a0438` [00:35:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#5590: Bump extensions/Wikibase from `c496ca4` to `5c27652` - 13https://github.com/miraheze/mediawiki/pull/5590 [00:35:46] [02mediawiki] 07dependabot[bot] opened pull request 03#5601: Bump extensions/GlobalPreferences from `2503bb0` to `08a0438` - 13https://github.com/miraheze/mediawiki/pull/5601 [00:35:48] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalPreferences-08a0438 - 13https://github.com/miraheze/mediawiki [00:35:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#5592: Bump extensions/CreateWiki from `8fa2186` to `931a7c7` - 13https://github.com/miraheze/mediawiki/pull/5592 [00:35:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#5595: Bump extensions/Disambiguator from `a6eeae4` to `55e0f01` - 13https://github.com/miraheze/mediawiki/pull/5595 [00:35:52] [02mediawiki] 07github-actions[bot] labeled pull request 03#5592: Bump extensions/CreateWiki from `8fa2186` to `931a7c7` - 13https://github.com/miraheze/mediawiki/pull/5592 [00:35:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#5594: Bump extensions/GlobalNewFiles from `68aca4a` to `dee5228` - 13https://github.com/miraheze/mediawiki/pull/5594 [00:35:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#5574: Bump extensions/BetaFeatures from `178df60` to `9f1c89f` - 13https://github.com/miraheze/mediawiki/pull/5574 [00:35:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#5592: Bump extensions/CreateWiki from `8fa2186` to `931a7c7` - 13https://github.com/miraheze/mediawiki/pull/5592 [00:35:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#5592: Bump extensions/CreateWiki from `8fa2186` to `931a7c7` - 13https://github.com/miraheze/mediawiki/pull/5592 [00:36:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#5593: Bump extensions/CiteThisPage from `bb4d47b` to `2972451` - 13https://github.com/miraheze/mediawiki/pull/5593 [00:36:01] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Mask-6565da0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e87bda535abb [00:36:03] [02miraheze/mediawiki] 07dependabot[bot] 03e87bda5 - Bump skins/Mask from `0b93fa7` to `6565da0` [00:36:04] [02mediawiki] 07dependabot[bot] opened pull request 03#5602: Bump skins/Mask from `0b93fa7` to `6565da0` - 13https://github.com/miraheze/mediawiki/pull/5602 [00:36:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Mask-6565da0 - 13https://github.com/miraheze/mediawiki [00:36:07] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/OAuth-6baeee2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d157be248e7a [00:36:09] [02miraheze/mediawiki] 07dependabot[bot] 03d157be2 - Bump extensions/OAuth from `187a76a` to `6baeee2` [00:36:10] [02mediawiki] 07dependabot[bot] opened pull request 03#5603: Bump extensions/OAuth from `187a76a` to `6baeee2` - 13https://github.com/miraheze/mediawiki/pull/5603 [00:36:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/OAuth-6baeee2 - 13https://github.com/miraheze/mediawiki [00:36:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/CologneBlue-11135a0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b0ed2d757e05 [00:36:15] [02miraheze/mediawiki] 07dependabot[bot] 03b0ed2d7 - Bump skins/CologneBlue from `6f1b3c6` to `11135a0` [00:36:16] [02mediawiki] 07dependabot[bot] opened pull request 03#5604: Bump skins/CologneBlue from `6f1b3c6` to `11135a0` - 13https://github.com/miraheze/mediawiki/pull/5604 [00:36:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/CologneBlue-11135a0 - 13https://github.com/miraheze/mediawiki [00:36:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#5596: Bump extensions/RSS from `e76c5a6` to `f0b82fb` - 13https://github.com/miraheze/mediawiki/pull/5596 [00:36:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikiEditor-6cde9e7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ba0ada4dadf2 [00:36:22] [02miraheze/mediawiki] 07dependabot[bot] 03ba0ada4 - Bump extensions/WikiEditor from `3945b36` to `6cde9e7` [00:36:24] [02mediawiki] 07dependabot[bot] opened pull request 03#5605: Bump extensions/WikiEditor from `3945b36` to `6cde9e7` - 13https://github.com/miraheze/mediawiki/pull/5605 [00:36:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikiEditor-6cde9e7 - 13https://github.com/miraheze/mediawiki [00:36:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Cite-293a854 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/aba1513dee7f [00:36:28] [02miraheze/mediawiki] 07dependabot[bot] 03aba1513 - Bump extensions/Cite from `2f9c294` to `293a854` [00:36:30] [02mediawiki] 07dependabot[bot] opened pull request 03#5606: Bump extensions/Cite from `2f9c294` to `293a854` - 13https://github.com/miraheze/mediawiki/pull/5606 [00:36:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Cite-293a854 - 13https://github.com/miraheze/mediawiki [00:36:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#5597: Bump extensions/MultimediaViewer from `ea6584d` to `7d5845f` - 13https://github.com/miraheze/mediawiki/pull/5597 [00:36:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#5598: Bump extensions/CountDownClock from `451d486` to `8674d6c` - 13https://github.com/miraheze/mediawiki/pull/5598 [00:36:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TemplateData-3489558 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8ea8dcc11097 [00:36:37] [02miraheze/mediawiki] 07dependabot[bot] 038ea8dcc - Bump extensions/TemplateData from `e7dbc90` to `3489558` [00:36:39] [02mediawiki] 07dependabot[bot] opened pull request 03#5607: Bump extensions/TemplateData from `e7dbc90` to `3489558` - 13https://github.com/miraheze/mediawiki/pull/5607 [00:36:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TemplateData-3489558 - 13https://github.com/miraheze/mediawiki [00:36:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageTriage-914c4b1 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/2d6aaf1e8b86 [00:36:43] [02miraheze/mediawiki] 07dependabot[bot] 032d6aaf1 - Bump extensions/PageTriage from `369599b` to `914c4b1` [00:36:45] [02mediawiki] 07dependabot[bot] opened pull request 03#5608: Bump extensions/PageTriage from `369599b` to `914c4b1` - 13https://github.com/miraheze/mediawiki/pull/5608 [00:36:46] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageTriage-914c4b1 - 13https://github.com/miraheze/mediawiki [00:36:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/QuizGame-9ea8dfa [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8dbb0dd82fc0 [00:36:49] [02miraheze/mediawiki] 07dependabot[bot] 038dbb0dd - Bump extensions/QuizGame from `9b43e02` to `9ea8dfa` [00:36:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#5600: Bump extensions/Linter from `f0c9993` to `5b70116` - 13https://github.com/miraheze/mediawiki/pull/5600 [00:36:52] [02mediawiki] 07dependabot[bot] opened pull request 03#5609: Bump extensions/QuizGame from `9b43e02` to `9ea8dfa` - 13https://github.com/miraheze/mediawiki/pull/5609 [00:36:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#5599: Bump extensions/MobileFrontend from `5d3c0b8` to `6222894` - 13https://github.com/miraheze/mediawiki/pull/5599 [00:36:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/QuizGame-9ea8dfa - 13https://github.com/miraheze/mediawiki [00:36:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#5602: Bump skins/Mask from `0b93fa7` to `6565da0` - 13https://github.com/miraheze/mediawiki/pull/5602 [00:36:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#5603: Bump extensions/OAuth from `187a76a` to `6baeee2` - 13https://github.com/miraheze/mediawiki/pull/5603 [00:37:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/RemovePII-b029250 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7131bb48e7c8 [00:37:01] [02miraheze/mediawiki] 07dependabot[bot] 037131bb4 - Bump extensions/RemovePII from `7491aba` to `b029250` [00:37:03] [02mediawiki] 07dependabot[bot] opened pull request 03#5610: Bump extensions/RemovePII from `7491aba` to `b029250` - 13https://github.com/miraheze/mediawiki/pull/5610 [00:37:04] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/RemovePII-b029250 - 13https://github.com/miraheze/mediawiki [00:37:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CollapsibleVector-eccc863 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/039762cea529 [00:37:07] [02miraheze/mediawiki] 07dependabot[bot] 03039762c - Bump extensions/CollapsibleVector from `202943b` to `eccc863` [00:37:09] [02mediawiki] 07dependabot[bot] opened pull request 03#5611: Bump extensions/CollapsibleVector from `202943b` to `eccc863` - 13https://github.com/miraheze/mediawiki/pull/5611 [00:37:10] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CollapsibleVector-eccc863 - 13https://github.com/miraheze/mediawiki [00:37:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/apex-5448eaf [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d8b82996a284 [00:37:13] [02miraheze/mediawiki] 07dependabot[bot] 03d8b8299 - Bump skins/apex from `3297eb5` to `5448eaf` [00:37:15] [02mediawiki] 07dependabot[bot] opened pull request 03#5612: Bump skins/apex from `3297eb5` to `5448eaf` - 13https://github.com/miraheze/mediawiki/pull/5612 [00:37:15] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.02, 9.56, 8.94 [00:37:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/apex-5448eaf - 13https://github.com/miraheze/mediawiki [00:37:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CentralAuth-306fa3a [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/59424680090d [00:37:19] [02miraheze/mediawiki] 07dependabot[bot] 035942468 - Bump extensions/CentralAuth from `8ce8907` to `306fa3a` [00:37:21] [02mediawiki] 07dependabot[bot] opened pull request 03#5613: Bump extensions/CentralAuth from `8ce8907` to `306fa3a` - 13https://github.com/miraheze/mediawiki/pull/5613 [00:37:22] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CentralAuth-306fa3a - 13https://github.com/miraheze/mediawiki [00:37:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/OATHAuth-5dfe3dd [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f2a58b14c755 [00:37:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.31, 8.01, 7.64 [00:37:25] [02miraheze/mediawiki] 07dependabot[bot] 03f2a58b1 - Bump extensions/OATHAuth from `1f0b9f9` to `5dfe3dd` [00:37:27] [02mediawiki] 07dependabot[bot] opened pull request 03#5614: Bump extensions/OATHAuth from `1f0b9f9` to `5dfe3dd` - 13https://github.com/miraheze/mediawiki/pull/5614 [00:37:28] [02mediawiki] 07github-actions[bot] labeled pull request 03#5604: Bump skins/CologneBlue from `6f1b3c6` to `11135a0` - 13https://github.com/miraheze/mediawiki/pull/5604 [00:37:30] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/OATHAuth-5dfe3dd - 13https://github.com/miraheze/mediawiki [00:37:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Anisa-5fdbdb4 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/e789b7bdec83 [00:37:33] [02miraheze/mediawiki] 07dependabot[bot] 03e789b7b - Bump skins/Anisa from `332e5ed` to `5fdbdb4` [00:37:34] [02mediawiki] 07dependabot[bot] opened pull request 03#5615: Bump skins/Anisa from `332e5ed` to `5fdbdb4` - 13https://github.com/miraheze/mediawiki/pull/5615 [00:37:36] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Anisa-5fdbdb4 - 13https://github.com/miraheze/mediawiki [00:37:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#5605: Bump extensions/WikiEditor from `3945b36` to `6cde9e7` - 13https://github.com/miraheze/mediawiki/pull/5605 [00:37:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AJAXPoll-cd322de [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/89e5585edc83 [00:37:40] [02miraheze/mediawiki] 07dependabot[bot] 0389e5585 - Bump extensions/AJAXPoll from `5f740ab` to `cd322de` [00:37:42] [02mediawiki] 07dependabot[bot] opened pull request 03#5616: Bump extensions/AJAXPoll from `5f740ab` to `cd322de` - 13https://github.com/miraheze/mediawiki/pull/5616 [00:37:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#5577: Bump extensions/LocalisationUpdate from `bed29e2` to `e5da971` - 13https://github.com/miraheze/mediawiki/pull/5577 [00:37:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AJAXPoll-cd322de - 13https://github.com/miraheze/mediawiki [00:37:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#5607: Bump extensions/TemplateData from `e7dbc90` to `3489558` - 13https://github.com/miraheze/mediawiki/pull/5607 [00:37:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WebChat-ac72c41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9c36c5cac58e [00:37:49] [02miraheze/mediawiki] 07dependabot[bot] 039c36c5c - Bump extensions/WebChat from `5475f20` to `ac72c41` [00:37:51] [02mediawiki] 07dependabot[bot] opened pull request 03#5617: Bump extensions/WebChat from `5475f20` to `ac72c41` - 13https://github.com/miraheze/mediawiki/pull/5617 [00:37:52] [02mediawiki] 07github-actions[bot] labeled pull request 03#5608: Bump extensions/PageTriage from `369599b` to `914c4b1` - 13https://github.com/miraheze/mediawiki/pull/5608 [00:37:54] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WebChat-ac72c41 - 13https://github.com/miraheze/mediawiki [00:37:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#5609: Bump extensions/QuizGame from `9b43e02` to `9ea8dfa` - 13https://github.com/miraheze/mediawiki/pull/5609 [00:37:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#5609: Bump extensions/QuizGame from `9b43e02` to `9ea8dfa` - 13https://github.com/miraheze/mediawiki/pull/5609 [00:37:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/wikihiero-01162f2 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c78ef07152f3 [00:38:00] [02miraheze/mediawiki] 07dependabot[bot] 03c78ef07 - Bump extensions/wikihiero from `ee8d1a2` to `01162f2` [00:38:01] [02mediawiki] 07dependabot[bot] opened pull request 03#5618: Bump extensions/wikihiero from `ee8d1a2` to `01162f2` - 13https://github.com/miraheze/mediawiki/pull/5618 [00:38:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/wikihiero-01162f2 - 13https://github.com/miraheze/mediawiki [00:38:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DPLForum-e2bd3fe [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/06d54b2bc4d0 [00:38:06] [02miraheze/mediawiki] 07dependabot[bot] 0306d54b2 - Bump extensions/DPLForum from `0e5f246` to `e2bd3fe` [00:38:07] [02mediawiki] 07dependabot[bot] opened pull request 03#5619: Bump extensions/DPLForum from `0e5f246` to `e2bd3fe` - 13https://github.com/miraheze/mediawiki/pull/5619 [00:38:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DPLForum-e2bd3fe - 13https://github.com/miraheze/mediawiki [00:38:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#5610: Bump extensions/RemovePII from `7491aba` to `b029250` - 13https://github.com/miraheze/mediawiki/pull/5610 [00:38:12] [02mediawiki] 07github-actions[bot] labeled pull request 03#5610: Bump extensions/RemovePII from `7491aba` to `b029250` - 13https://github.com/miraheze/mediawiki/pull/5610 [00:38:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Citizen-3e6a240 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/105925d37a9a [00:38:15] [02miraheze/mediawiki] 07dependabot[bot] 03105925d - Bump skins/Citizen from `2ca1742` to `3e6a240` [00:38:16] [02mediawiki] 07dependabot[bot] opened pull request 03#5620: Bump skins/Citizen from `2ca1742` to `3e6a240` - 13https://github.com/miraheze/mediawiki/pull/5620 [00:38:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Citizen-3e6a240 - 13https://github.com/miraheze/mediawiki [00:38:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#5611: Bump extensions/CollapsibleVector from `202943b` to `eccc863` - 13https://github.com/miraheze/mediawiki/pull/5611 [00:38:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Timeless-a942660 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c25fcda3530d [00:38:22] [02miraheze/mediawiki] 07dependabot[bot] 03c25fcda - Bump skins/Timeless from `6f32ef0` to `a942660` [00:38:24] [02mediawiki] 07dependabot[bot] opened pull request 03#5621: Bump skins/Timeless from `6f32ef0` to `a942660` - 13https://github.com/miraheze/mediawiki/pull/5621 [00:38:25] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Timeless-a942660 - 13https://github.com/miraheze/mediawiki [00:38:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Cargo-0f06913 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/6e9804bdfba8 [00:38:28] [02miraheze/mediawiki] 07dependabot[bot] 036e9804b - Bump extensions/Cargo from `dc101b6` to `0f06913` [00:38:30] [02mediawiki] 07dependabot[bot] opened pull request 03#5622: Bump extensions/Cargo from `dc101b6` to `0f06913` - 13https://github.com/miraheze/mediawiki/pull/5622 [00:38:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Cargo-0f06913 - 13https://github.com/miraheze/mediawiki [00:38:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#5614: Bump extensions/OATHAuth from `1f0b9f9` to `5dfe3dd` - 13https://github.com/miraheze/mediawiki/pull/5614 [00:38:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#5614: Bump extensions/OATHAuth from `1f0b9f9` to `5dfe3dd` - 13https://github.com/miraheze/mediawiki/pull/5614 [00:38:36] [02mediawiki] 07github-actions[bot] labeled pull request 03#5616: Bump extensions/AJAXPoll from `5f740ab` to `cd322de` - 13https://github.com/miraheze/mediawiki/pull/5616 [00:38:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#5615: Bump skins/Anisa from `332e5ed` to `5fdbdb4` - 13https://github.com/miraheze/mediawiki/pull/5615 [00:38:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#5617: Bump extensions/WebChat from `5475f20` to `ac72c41` - 13https://github.com/miraheze/mediawiki/pull/5617 [00:38:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/IncidentReporting-39c4b13 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/fa1f282fa19d [00:38:42] [02miraheze/mediawiki] 07dependabot[bot] 03fa1f282 - Bump extensions/IncidentReporting from `54dfbd5` to `39c4b13` [00:38:43] [02mediawiki] 07dependabot[bot] opened pull request 03#5623: Bump extensions/IncidentReporting from `54dfbd5` to `39c4b13` - 13https://github.com/miraheze/mediawiki/pull/5623 [00:38:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/IncidentReporting-39c4b13 - 13https://github.com/miraheze/mediawiki [00:38:46] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalCssJs-e8af277 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/834376361b43 [00:38:48] [02miraheze/mediawiki] 07dependabot[bot] 038343763 - Bump extensions/GlobalCssJs from `958a04c` to `e8af277` [00:38:49] [02mediawiki] 07dependabot[bot] opened pull request 03#5624: Bump extensions/GlobalCssJs from `958a04c` to `e8af277` - 13https://github.com/miraheze/mediawiki/pull/5624 [00:38:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#5619: Bump extensions/DPLForum from `0e5f246` to `e2bd3fe` - 13https://github.com/miraheze/mediawiki/pull/5619 [00:38:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalCssJs-e8af277 - 13https://github.com/miraheze/mediawiki [00:38:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#5551: Bump extensions/UniversalLanguageSelector from `0bd9a68` to `32ee5f5` - 13https://github.com/miraheze/mediawiki/pull/5551 [00:38:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#5551: Bump extensions/UniversalLanguageSelector from `0bd9a68` to `32ee5f5` - 13https://github.com/miraheze/mediawiki/pull/5551 [00:38:57] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Comments-ad12023 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/64ffffa918d9 [00:38:58] [02miraheze/mediawiki] 07dependabot[bot] 0364ffffa - Bump extensions/Comments from `72ce95c` to `ad12023` [00:39:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#5613: Bump extensions/CentralAuth from `8ce8907` to `306fa3a` - 13https://github.com/miraheze/mediawiki/pull/5613 [00:39:01] [02mediawiki] 07dependabot[bot] opened pull request 03#5625: Bump extensions/Comments from `72ce95c` to `ad12023` - 13https://github.com/miraheze/mediawiki/pull/5625 [00:39:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Comments-ad12023 - 13https://github.com/miraheze/mediawiki [00:39:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ManageWiki-4635bbf [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/26dbb38f5366 [00:39:06] [02miraheze/mediawiki] 07dependabot[bot] 0326dbb38 - Bump extensions/ManageWiki from `35b5466` to `4635bbf` [00:39:07] [02mediawiki] 07dependabot[bot] opened pull request 03#5626: Bump extensions/ManageWiki from `35b5466` to `4635bbf` - 13https://github.com/miraheze/mediawiki/pull/5626 [00:39:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ManageWiki-4635bbf - 13https://github.com/miraheze/mediawiki [00:39:10] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/DuskToDawn-b0aed00 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5cb918755082 [00:39:12] [02miraheze/mediawiki] 07dependabot[bot] 035cb9187 - Bump skins/DuskToDawn from `f2aa1c8` to `b0aed00` [00:39:13] [02mediawiki] 07dependabot[bot] opened pull request 03#5627: Bump skins/DuskToDawn from `f2aa1c8` to `b0aed00` - 13https://github.com/miraheze/mediawiki/pull/5627 [00:39:14] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.46, 9.41, 8.96 [00:39:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/DuskToDawn-b0aed00 - 13https://github.com/miraheze/mediawiki [00:39:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalUserPage-d4f69ec [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/c66c2d73d56a [00:39:18] [02miraheze/mediawiki] 07dependabot[bot] 03c66c2d7 - Bump extensions/GlobalUserPage from `062507f` to `d4f69ec` [00:39:20] [02mediawiki] 07dependabot[bot] opened pull request 03#5628: Bump extensions/GlobalUserPage from `062507f` to `d4f69ec` - 13https://github.com/miraheze/mediawiki/pull/5628 [00:39:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalUserPage-d4f69ec - 13https://github.com/miraheze/mediawiki [00:39:22] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Gadgets-184e601 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7628fd9edacf [00:39:24] [02miraheze/mediawiki] 07dependabot[bot] 037628fd9 - Bump extensions/Gadgets from `3e5138f` to `184e601` [00:39:25] [02mediawiki] 07dependabot[bot] opened pull request 03#5629: Bump extensions/Gadgets from `3e5138f` to `184e601` - 13https://github.com/miraheze/mediawiki/pull/5629 [00:39:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Gadgets-184e601 - 13https://github.com/miraheze/mediawiki [00:39:28] [02mediawiki] 07github-actions[bot] labeled pull request 03#5622: Bump extensions/Cargo from `dc101b6` to `0f06913` - 13https://github.com/miraheze/mediawiki/pull/5622 [00:39:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#5620: Bump skins/Citizen from `2ca1742` to `3e6a240` - 13https://github.com/miraheze/mediawiki/pull/5620 [00:39:31] [02mediawiki] 07github-actions[bot] labeled pull request 03#5620: Bump skins/Citizen from `2ca1742` to `3e6a240` - 13https://github.com/miraheze/mediawiki/pull/5620 [00:39:33] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Metrolook-49b0507 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/d7f40977cf92 [00:39:34] [02miraheze/mediawiki] 07dependabot[bot] 03d7f4097 - Bump skins/Metrolook from `ba638aa` to `49b0507` [00:39:36] [02mediawiki] 07dependabot[bot] opened pull request 03#5630: Bump skins/Metrolook from `ba638aa` to `49b0507` - 13https://github.com/miraheze/mediawiki/pull/5630 [00:39:37] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Metrolook-49b0507 - 13https://github.com/miraheze/mediawiki [00:39:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DataTransfer-690b308 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/19c3eb1d5712 [00:39:40] [02miraheze/mediawiki] 07dependabot[bot] 0319c3eb1 - Bump extensions/DataTransfer from `68a4755` to `690b308` [00:39:42] [02mediawiki] 07dependabot[bot] opened pull request 03#5631: Bump extensions/DataTransfer from `68a4755` to `690b308` - 13https://github.com/miraheze/mediawiki/pull/5631 [00:39:43] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DataTransfer-690b308 - 13https://github.com/miraheze/mediawiki [00:39:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Math-e415034 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b2400321ed68 [00:39:46] [02miraheze/mediawiki] 07dependabot[bot] 03b240032 - Bump extensions/Math from `83a2b7a` to `e415034` [00:39:48] [02mediawiki] 07dependabot[bot] opened pull request 03#5632: Bump extensions/Math from `83a2b7a` to `e415034` - 13https://github.com/miraheze/mediawiki/pull/5632 [00:39:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Math-e415034 - 13https://github.com/miraheze/mediawiki [00:39:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#5623: Bump extensions/IncidentReporting from `54dfbd5` to `39c4b13` - 13https://github.com/miraheze/mediawiki/pull/5623 [00:39:52] [02mediawiki] 07github-actions[bot] labeled pull request 03#5623: Bump extensions/IncidentReporting from `54dfbd5` to `39c4b13` - 13https://github.com/miraheze/mediawiki/pull/5623 [00:39:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#5624: Bump extensions/GlobalCssJs from `958a04c` to `e8af277` - 13https://github.com/miraheze/mediawiki/pull/5624 [00:39:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Interwiki-499a15d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/691d0c815509 [00:39:57] [02miraheze/mediawiki] 07dependabot[bot] 03691d0c8 - Bump extensions/Interwiki from `3a5d667` to `499a15d` [00:39:59] [02mediawiki] 07dependabot[bot] opened pull request 03#5633: Bump extensions/Interwiki from `3a5d667` to `499a15d` - 13https://github.com/miraheze/mediawiki/pull/5633 [00:40:00] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Interwiki-499a15d - 13https://github.com/miraheze/mediawiki [00:40:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#5625: Bump extensions/Comments from `72ce95c` to `ad12023` - 13https://github.com/miraheze/mediawiki/pull/5625 [00:40:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Gamepress-6b68f32 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4449f66dca0f [00:40:05] [02miraheze/mediawiki] 07dependabot[bot] 034449f66 - Bump skins/Gamepress from `5ab4cf5` to `6b68f32` [00:40:06] [02mediawiki] 07dependabot[bot] opened pull request 03#5634: Bump skins/Gamepress from `5ab4cf5` to `6b68f32` - 13https://github.com/miraheze/mediawiki/pull/5634 [00:40:08] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Gamepress-6b68f32 - 13https://github.com/miraheze/mediawiki [00:40:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/BlogPage-0b24c44 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/4bf04f3bd181 [00:40:11] [02miraheze/mediawiki] 07dependabot[bot] 034bf04f3 - Bump extensions/BlogPage from `05a8163` to `0b24c44` [00:40:12] [02mediawiki] 07dependabot[bot] opened pull request 03#5635: Bump extensions/BlogPage from `05a8163` to `0b24c44` - 13https://github.com/miraheze/mediawiki/pull/5635 [00:40:14] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/BlogPage-0b24c44 - 13https://github.com/miraheze/mediawiki [00:40:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#5627: Bump skins/DuskToDawn from `f2aa1c8` to `b0aed00` - 13https://github.com/miraheze/mediawiki/pull/5627 [00:40:17] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Echo-7501007 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ddf472d1a881 [00:40:18] [02miraheze/mediawiki] 07dependabot[bot] 03ddf472d - Bump extensions/Echo from `372639d` to `7501007` [00:40:20] [02mediawiki] 07dependabot[bot] opened pull request 03#5636: Bump extensions/Echo from `372639d` to `7501007` - 13https://github.com/miraheze/mediawiki/pull/5636 [00:40:21] [02mediawiki] 07github-actions[bot] labeled pull request 03#5629: Bump extensions/Gadgets from `3e5138f` to `184e601` - 13https://github.com/miraheze/mediawiki/pull/5629 [00:40:23] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Echo-7501007 - 13https://github.com/miraheze/mediawiki [00:40:24] [02mediawiki] 07github-actions[bot] labeled pull request 03#5630: Bump skins/Metrolook from `ba638aa` to `49b0507` - 13https://github.com/miraheze/mediawiki/pull/5630 [00:40:26] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/InputBox-e6f8ca0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/22b05772bea9 [00:40:27] [02miraheze/mediawiki] 07dependabot[bot] 0322b0577 - Bump extensions/InputBox from `02cf4c4` to `e6f8ca0` [00:40:29] [02mediawiki] 07dependabot[bot] opened pull request 03#5637: Bump extensions/InputBox from `02cf4c4` to `e6f8ca0` - 13https://github.com/miraheze/mediawiki/pull/5637 [00:40:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#5631: Bump extensions/DataTransfer from `68a4755` to `690b308` - 13https://github.com/miraheze/mediawiki/pull/5631 [00:40:32] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/InputBox-e6f8ca0 - 13https://github.com/miraheze/mediawiki [00:40:33] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SocialProfile-f1d6a5b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/0dbc51a2d7a9 [00:40:35] [02miraheze/mediawiki] 07dependabot[bot] 030dbc51a - Bump extensions/SocialProfile from `37838f0` to `f1d6a5b` [00:40:36] [02mediawiki] 07dependabot[bot] opened pull request 03#5638: Bump extensions/SocialProfile from `37838f0` to `f1d6a5b` - 13https://github.com/miraheze/mediawiki/pull/5638 [00:40:38] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SocialProfile-f1d6a5b - 13https://github.com/miraheze/mediawiki [00:40:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Popups-7206c35 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7a2e5e5e7674 [00:40:41] [02miraheze/mediawiki] 07dependabot[bot] 037a2e5e5 - Bump extensions/Popups from `3b9d419` to `7206c35` [00:40:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Popups-7206c35 - 13https://github.com/miraheze/mediawiki [00:40:44] [02mediawiki] 07dependabot[bot] opened pull request 03#5639: Bump extensions/Popups from `3b9d419` to `7206c35` - 13https://github.com/miraheze/mediawiki/pull/5639 [00:40:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ApprovedRevs-bd12cb0 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ac7fc066cb98 [00:40:47] [02miraheze/mediawiki] 07dependabot[bot] 03ac7fc06 - Bump extensions/ApprovedRevs from `a7fb6ac` to `bd12cb0` [00:40:48] [02mediawiki] 07dependabot[bot] opened pull request 03#5640: Bump extensions/ApprovedRevs from `a7fb6ac` to `bd12cb0` - 13https://github.com/miraheze/mediawiki/pull/5640 [00:40:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ApprovedRevs-bd12cb0 - 13https://github.com/miraheze/mediawiki [00:40:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DataDump-88351b6 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/65373bbb2fe6 [00:40:53] [02miraheze/mediawiki] 07dependabot[bot] 0365373bb - Bump extensions/DataDump from `3d78106` to `88351b6` [00:40:54] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DataDump-88351b6 - 13https://github.com/miraheze/mediawiki [00:40:56] [02mediawiki] 07dependabot[bot] opened pull request 03#5641: Bump extensions/DataDump from `3d78106` to `88351b6` - 13https://github.com/miraheze/mediawiki/pull/5641 [00:40:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#5633: Bump extensions/Interwiki from `3a5d667` to `499a15d` - 13https://github.com/miraheze/mediawiki/pull/5633 [00:40:59] [02mediawiki] 07github-actions[bot] labeled pull request 03#5632: Bump extensions/Math from `83a2b7a` to `e415034` - 13https://github.com/miraheze/mediawiki/pull/5632 [00:41:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#5601: Bump extensions/GlobalPreferences from `2503bb0` to `08a0438` - 13https://github.com/miraheze/mediawiki/pull/5601 [00:41:02] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TimedMediaHandler-4899bda [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/9f1b33f240f1 [00:41:03] [02miraheze/mediawiki] 07dependabot[bot] 039f1b33f - Bump extensions/TimedMediaHandler from `c455436` to `4899bda` [00:41:05] [02mediawiki] 07dependabot[bot] opened pull request 03#5642: Bump extensions/TimedMediaHandler from `c455436` to `4899bda` - 13https://github.com/miraheze/mediawiki/pull/5642 [00:41:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TimedMediaHandler-4899bda - 13https://github.com/miraheze/mediawiki [00:41:08] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AntiSpoof-44f4504 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/38d6ea557c54 [00:41:09] [02miraheze/mediawiki] 07dependabot[bot] 0338d6ea5 - Bump extensions/AntiSpoof from `068e62b` to `44f4504` [00:41:11] [02mediawiki] 07dependabot[bot] opened pull request 03#5643: Bump extensions/AntiSpoof from `068e62b` to `44f4504` - 13https://github.com/miraheze/mediawiki/pull/5643 [00:41:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AntiSpoof-44f4504 - 13https://github.com/miraheze/mediawiki [00:41:14] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DynamicPageList3-b36e98f [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/cbdc657716e0 [00:41:15] [02miraheze/mediawiki] 07dependabot[bot] 03cbdc657 - Bump extensions/DynamicPageList3 from `db51e49` to `b36e98f` [00:41:17] [02mediawiki] 07dependabot[bot] opened pull request 03#5644: Bump extensions/DynamicPageList3 from `db51e49` to `b36e98f` - 13https://github.com/miraheze/mediawiki/pull/5644 [00:41:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DynamicPageList3-b36e98f - 13https://github.com/miraheze/mediawiki [00:41:20] [02mediawiki] 07github-actions[bot] labeled pull request 03#5634: Bump skins/Gamepress from `5ab4cf5` to `6b68f32` - 13https://github.com/miraheze/mediawiki/pull/5634 [00:41:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Evelution-732289b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/5a8f466cf476 [00:41:23] [02miraheze/mediawiki] 07dependabot[bot] 035a8f466 - Bump skins/Evelution from `73acf5f` to `732289b` [00:41:24] [02mediawiki] 07dependabot[bot] opened pull request 03#5645: Bump skins/Evelution from `73acf5f` to `732289b` - 13https://github.com/miraheze/mediawiki/pull/5645 [00:41:26] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Evelution-732289b - 13https://github.com/miraheze/mediawiki [00:41:27] [02mediawiki] 07github-actions[bot] labeled pull request 03#5635: Bump extensions/BlogPage from `05a8163` to `0b24c44` - 13https://github.com/miraheze/mediawiki/pull/5635 [00:41:29] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/VoteNY-8c17948 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/62ef0f198530 [00:41:30] [02miraheze/mediawiki] 07dependabot[bot] 0362ef0f1 - Bump extensions/VoteNY from `3015dd7` to `8c17948` [00:41:32] [02mediawiki] 07github-actions[bot] labeled pull request 03#5638: Bump extensions/SocialProfile from `37838f0` to `f1d6a5b` - 13https://github.com/miraheze/mediawiki/pull/5638 [00:41:33] [02mediawiki] 07dependabot[bot] opened pull request 03#5646: Bump extensions/VoteNY from `3015dd7` to `8c17948` - 13https://github.com/miraheze/mediawiki/pull/5646 [00:41:35] [02mediawiki] 07github-actions[bot] labeled pull request 03#5636: Bump extensions/Echo from `372639d` to `7501007` - 13https://github.com/miraheze/mediawiki/pull/5636 [00:41:36] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/VoteNY-8c17948 - 13https://github.com/miraheze/mediawiki [00:41:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Vector-28e5d64 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/8cd99b640735 [00:41:39] [02miraheze/mediawiki] 07dependabot[bot] 038cd99b6 - Bump skins/Vector from `7908b86` to `28e5d64` [00:41:41] [02mediawiki] 07dependabot[bot] opened pull request 03#5647: Bump skins/Vector from `7908b86` to `28e5d64` - 13https://github.com/miraheze/mediawiki/pull/5647 [00:41:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Vector-28e5d64 - 13https://github.com/miraheze/mediawiki [00:41:44] [02mediawiki] 07github-actions[bot] labeled pull request 03#5639: Bump extensions/Popups from `3b9d419` to `7206c35` - 13https://github.com/miraheze/mediawiki/pull/5639 [00:41:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Flow-c4c9f1c [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/073ac3c2ba20 [00:41:47] [02miraheze/mediawiki] 07dependabot[bot] 03073ac3c - Bump extensions/Flow from `05c7119` to `c4c9f1c` [00:41:48] [02mediawiki] 07dependabot[bot] opened pull request 03#5648: Bump extensions/Flow from `05c7119` to `c4c9f1c` - 13https://github.com/miraheze/mediawiki/pull/5648 [00:41:50] [02mediawiki] 07github-actions[bot] labeled pull request 03#5641: Bump extensions/DataDump from `3d78106` to `88351b6` - 13https://github.com/miraheze/mediawiki/pull/5641 [00:41:51] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Flow-c4c9f1c - 13https://github.com/miraheze/mediawiki [00:41:53] [02mediawiki] 07github-actions[bot] labeled pull request 03#5642: Bump extensions/TimedMediaHandler from `c455436` to `4899bda` - 13https://github.com/miraheze/mediawiki/pull/5642 [00:41:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Nuke-13a48d4 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/965fc5876b43 [00:41:56] [02miraheze/mediawiki] 07dependabot[bot] 03965fc58 - Bump extensions/Nuke from `df82691` to `13a48d4` [00:41:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#5640: Bump extensions/ApprovedRevs from `a7fb6ac` to `bd12cb0` - 13https://github.com/miraheze/mediawiki/pull/5640 [00:41:59] [02mediawiki] 07dependabot[bot] opened pull request 03#5649: Bump extensions/Nuke from `df82691` to `13a48d4` - 13https://github.com/miraheze/mediawiki/pull/5649 [00:42:00] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Nuke-13a48d4 - 13https://github.com/miraheze/mediawiki [00:42:02] [02mediawiki] 07github-actions[bot] labeled pull request 03#5643: Bump extensions/AntiSpoof from `068e62b` to `44f4504` - 13https://github.com/miraheze/mediawiki/pull/5643 [00:42:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Kartographer-396d49b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b3f6cd0d0fed [00:42:05] [02miraheze/mediawiki] 07dependabot[bot] 03b3f6cd0 - Bump extensions/Kartographer from `d84bfc0` to `396d49b` [00:42:06] [02mediawiki] 07dependabot[bot] opened pull request 03#5650: Bump extensions/Kartographer from `d84bfc0` to `396d49b` - 13https://github.com/miraheze/mediawiki/pull/5650 [00:42:08] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Kartographer-396d49b - 13https://github.com/miraheze/mediawiki [00:42:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CheckUser-c147485 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ff36efd82d86 [00:42:11] [02miraheze/mediawiki] 07dependabot[bot] 03ff36efd - Bump extensions/CheckUser from `b434f05` to `c147485` [00:42:12] [02mediawiki] 07github-actions[bot] labeled pull request 03#5644: Bump extensions/DynamicPageList3 from `db51e49` to `b36e98f` - 13https://github.com/miraheze/mediawiki/pull/5644 [00:42:14] [02mediawiki] 07dependabot[bot] opened pull request 03#5651: Bump extensions/CheckUser from `b434f05` to `c147485` - 13https://github.com/miraheze/mediawiki/pull/5651 [00:42:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#5644: Bump extensions/DynamicPageList3 from `db51e49` to `b36e98f` - 13https://github.com/miraheze/mediawiki/pull/5644 [00:42:17] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CheckUser-c147485 - 13https://github.com/miraheze/mediawiki [00:42:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AdminLinks-108fec7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/f8e9d36d2a80 [00:42:20] [02miraheze/mediawiki] 07dependabot[bot] 03f8e9d36 - Bump extensions/AdminLinks from `0180867` to `108fec7` [00:42:21] [02mediawiki] 07dependabot[bot] opened pull request 03#5652: Bump extensions/AdminLinks from `0180867` to `108fec7` - 13https://github.com/miraheze/mediawiki/pull/5652 [00:42:23] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AdminLinks-108fec7 - 13https://github.com/miraheze/mediawiki [00:42:24] [02mediawiki] 07github-actions[bot] labeled pull request 03#5646: Bump extensions/VoteNY from `3015dd7` to `8c17948` - 13https://github.com/miraheze/mediawiki/pull/5646 [00:42:26] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Renameuser-4e8eb85 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/bb8f9f0f7195 [00:42:27] [02miraheze/mediawiki] 07dependabot[bot] 03bb8f9f0 - Bump extensions/Renameuser from `0f598d8` to `4e8eb85` [00:42:29] [02mediawiki] 07dependabot[bot] opened pull request 03#5653: Bump extensions/Renameuser from `0f598d8` to `4e8eb85` - 13https://github.com/miraheze/mediawiki/pull/5653 [00:42:30] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Renameuser-4e8eb85 - 13https://github.com/miraheze/mediawiki [00:42:32] [02mediawiki] 07github-actions[bot] labeled pull request 03#5647: Bump skins/Vector from `7908b86` to `28e5d64` - 13https://github.com/miraheze/mediawiki/pull/5647 [00:42:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#5645: Bump skins/Evelution from `73acf5f` to `732289b` - 13https://github.com/miraheze/mediawiki/pull/5645 [00:42:35] [02mediawiki] 07github-actions[bot] labeled pull request 03#5626: Bump extensions/ManageWiki from `35b5466` to `4635bbf` - 13https://github.com/miraheze/mediawiki/pull/5626 [00:42:36] [02mediawiki] 07github-actions[bot] labeled pull request 03#5626: Bump extensions/ManageWiki from `35b5466` to `4635bbf` - 13https://github.com/miraheze/mediawiki/pull/5626 [00:42:38] [02mediawiki] 07github-actions[bot] labeled pull request 03#5626: Bump extensions/ManageWiki from `35b5466` to `4635bbf` - 13https://github.com/miraheze/mediawiki/pull/5626 [00:42:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#5628: Bump extensions/GlobalUserPage from `062507f` to `d4f69ec` - 13https://github.com/miraheze/mediawiki/pull/5628 [00:42:41] [02mediawiki] 07github-actions[bot] labeled pull request 03#5649: Bump extensions/Nuke from `df82691` to `13a48d4` - 13https://github.com/miraheze/mediawiki/pull/5649 [00:42:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#5618: Bump extensions/wikihiero from `ee8d1a2` to `01162f2` - 13https://github.com/miraheze/mediawiki/pull/5618 [00:42:44] [02mediawiki] 07github-actions[bot] labeled pull request 03#5650: Bump extensions/Kartographer from `d84bfc0` to `396d49b` - 13https://github.com/miraheze/mediawiki/pull/5650 [00:42:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#5651: Bump extensions/CheckUser from `b434f05` to `c147485` - 13https://github.com/miraheze/mediawiki/pull/5651 [00:42:47] [02mediawiki] 07github-actions[bot] labeled pull request 03#5652: Bump extensions/AdminLinks from `0180867` to `108fec7` - 13https://github.com/miraheze/mediawiki/pull/5652 [00:42:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#5648: Bump extensions/Flow from `05c7119` to `c4c9f1c` - 13https://github.com/miraheze/mediawiki/pull/5648 [00:42:50] [02mediawiki] 07github-actions[bot] labeled pull request 03#5653: Bump extensions/Renameuser from `0f598d8` to `4e8eb85` - 13https://github.com/miraheze/mediawiki/pull/5653 [00:42:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#5581: Bump extensions/CentralNotice from `3d946a0` to `f08dcd0` - 13https://github.com/miraheze/mediawiki/pull/5581 [00:42:53] [02mediawiki] 07github-actions[bot] labeled pull request 03#5612: Bump skins/apex from `3297eb5` to `5448eaf` - 13https://github.com/miraheze/mediawiki/pull/5612 [00:42:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#5637: Bump extensions/InputBox from `02cf4c4` to `e6f8ca0` - 13https://github.com/miraheze/mediawiki/pull/5637 [00:42:56] [02mediawiki] 07github-actions[bot] labeled pull request 03#5621: Bump skins/Timeless from `6f32ef0` to `a942660` - 13https://github.com/miraheze/mediawiki/pull/5621 [00:45:23] [02mediawiki] 07github-actions[bot] labeled pull request 03#5606: Bump extensions/Cite from `2f9c294` to `293a854` - 13https://github.com/miraheze/mediawiki/pull/5606 [00:49:10] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.41, 9.54, 9.12 [00:51:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.55, 9.38, 9.10 [00:55:33] db111's load is insane, nearing 160 [00:55:43] I can barely get any wikis on that database server to load [00:55:56] most requests just end up 502/503ing [01:00:25] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [01:01:05] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.42, 8.02, 8.47 [01:05:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.68, 8.45, 8.56 [01:06:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.17, 2.90 [01:07:01] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.46, 8.32, 8.50 [01:07:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:08:32] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.15, 2.98, 2.88 [01:11:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:14:01] [discord] @Orduin might you be around to restart `db111`? Load just hit 181 [01:14:07] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:14:14] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:15:24] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:15:31] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:15:36] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:15:41] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:15:51] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:16:11] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 7.222 second response time [01:16:17] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 5.938 second response time [01:16:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 4.00, 3.49, 3.13 [01:17:28] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.710 second response time [01:17:31] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22295 bytes in 1.559 second response time [01:17:38] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.015 second response time [01:17:51] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.319 second response time [01:18:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.98, 3.20, 3.07 [01:21:33] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 5.682 second response time [01:23:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.04, 1.96, 1.37 [01:25:33] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.239 second response time [01:25:51] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.56, 1.53, 1.28 [01:26:08] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.012 second response time [01:26:27] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:26:55] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.012 second response time [01:27:08] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:27:37] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 9.370 second response time [01:28:06] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.714 second response time [01:28:29] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 5.139 second response time [01:29:06] [discord] uhhhh what [01:29:09] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 1.216 second response time [01:30:29] RECOVERY - db111 APT on db111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [01:31:37] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2415 bytes in 0.272 second response time [01:31:58] !log db111 restart mariadb (400% CPU usage is very very bad) [01:31:59] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2415 bytes in 0.056 second response time [01:32:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:32:21] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.230 second response time [01:32:51] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.219 second response time [01:33:06] restart might not have fixed it [01:33:32] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.594 second response time [01:33:57] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.271 second response time [01:34:19] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.624 second response time [01:35:16] [discord] Yeah, I restated and we're already back at 400% CPU from ONE process [01:36:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:41:20] Created https://phabricator.miraheze.org/T8851 [01:41:20] [url] ⚓ T8851 db111 insanely high load | phabricator.miraheze.org [01:43:03] PROBLEM - db111 APT on db111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [01:43:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:44:47] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.30, 8.80, 7.80 [01:46:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.54, 8.50, 7.83 [01:50:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.73, 8.82, 8.17 [01:52:21] .op [01:52:21] Attempting to OP... [01:52:43] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.63, 9.22, 8.37 [01:54:42] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.80, 8.75, 8.32 [01:56:41] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.40, 8.07, 8.13 [02:05:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.50, 3.36, 2.82 [02:08:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.60, 8.16, 7.23 [02:09:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.16, 3.51, 3.01 [02:10:41] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.99, 7.82, 7.22 [02:11:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.37, 3.10, 2.91 [02:11:22] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.87, 2.84, 2.82 [02:18:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.32, 3.46, 3.13 [02:19:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.73, 9.28, 8.42 [02:19:46] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.42, 1.56, 1.25 [02:20:41] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.34, 3.00, 3.00 [02:25:27] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.68, 9.79, 8.98 [02:41:20] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.64, 7.61, 8.27 [02:45:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.99, 3.57, 3.08 [02:45:40] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.95, 8.11, 7.44 [02:47:09] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.16, 2.99, 2.93 [02:48:16] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.37, 9.09, 8.66 [02:50:15] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.65, 8.28, 8.44 [02:53:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.48, 8.10, 7.80 [02:53:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:58:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:02:09] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.193315506 secs [03:03:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.40, 2.15, 1.60 [03:04:16] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.0007894039154 secs [03:05:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.99, 1.70, 1.50 [03:13:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:16:18] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:18:16] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.758 second response time [03:21:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:26:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:28:02] PROBLEM - cp30 Stunnel HTTP for matomo101 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.233 second response time [03:28:07] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.011 second response time [03:28:09] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.252 second response time [03:28:23] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.015 second response time [03:29:37] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.011 second response time [03:32:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:44:39] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.38, 3.88, 2.31 [03:50:38] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.82, 1.86, 1.88 [03:56:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.71, 1.24, 1.62 [04:01:49] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.1224067211 secs [04:03:53] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.007701277733 secs [04:22:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:26:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:31:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:32:31] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:32:44] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:32:45] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:34:06] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:34:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:36:05] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.020 second response time [04:36:40] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 1.353 second response time [04:36:44] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.353 second response time [04:36:49] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22295 bytes in 0.360 second response time [04:42:32] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:42:33] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:42:46] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:43:03] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:43:27] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:44:50] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:44:59] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:45:47] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:46:19] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.1453947723 secs [04:47:47] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 6.440 second response time [04:48:49] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22295 bytes in 0.340 second response time [04:49:00] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.961 second response time [04:49:15] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.431 second response time [04:49:52] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 6.342 second response time [04:50:39] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22295 bytes in 1.100 second response time [04:50:57] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.017 second response time [04:51:11] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.307 second response time [04:52:32] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.004088819027 secs [05:01:44] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.19, 1.47, 1.99 [05:02:05] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.1176747084 secs [05:04:10] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.0110835433 secs [05:08:31] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.1445426047 secs [05:10:35] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.001061022282 secs [05:11:44] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.26, 1.38, 1.69 [05:22:13] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.1479299366 secs [05:24:17] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.003909647465 secs [05:24:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:31:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:37:52] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:37:54] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:39:47] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.305 second response time [05:39:51] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22295 bytes in 0.926 second response time [05:44:23] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.1311322153 secs [05:50:35] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.000615477562 secs [05:51:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:52:00] hrm, we have a recovery on `db111`, but I still can't load Special:Contributions on that wiki :( [05:52:47] MacFan4000, would killing the processes running on `db111` and rebooting the server help? [05:58:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:59:06] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.3014861643 secs [06:01:51] dmehus: I don’t have access to db servers so nothing I can do [06:02:19] MacFan4000, yeah... :( [06:02:37] hopefully Reception123 can take a look when he's around [06:02:45] Not something fun to wake up to, though... [06:03:02] I’m sure Infra will look when they see the UBN task [06:05:23] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.02221906185 secs [06:09:41] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.2233086228 secs [06:11:25] PROBLEM - db111 MariaDB on db111 is CRITICAL: Too many connections [06:11:47] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.01713186502 secs [06:15:25] RECOVERY - db111 MariaDB on db111 is OK: Uptime: 17012 Threads: 475 Questions: 8318202 Slow queries: 280442 Opens: 24711 Open tables: 16119 Queries per second avg: 488.960 [06:18:11] oof [06:18:20] that doesn't sound good [06:18:28] 280,442 slow queries? [06:20:16] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.215293169 secs [06:22:22] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.005470275879 secs [06:30:57] PROBLEM - db111 MariaDB on db111 is CRITICAL: Too many connections [06:32:52] RECOVERY - db111 MariaDB on db111 is OK: Uptime: 18059 Threads: 495 Questions: 8536347 Slow queries: 290420 Opens: 25753 Open tables: 16119 Queries per second avg: 472.692 [06:33:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:35:40] ergh this is insane [06:35:46] how is it showing recovery? [06:35:59] slow queries, and threads are increasing [06:38:27] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.222158134 secs [06:42:36] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.0009624361992 secs [06:43:45] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [06:48:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:53:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:58:44] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.1711244285 secs [07:00:49] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.001735895872 secs [07:09:45] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:27:35] PROBLEM - db111 MariaDB on db111 is CRITICAL: Too many connections [07:29:29] RECOVERY - db111 MariaDB on db111 is OK: Uptime: 21456 Threads: 498 Questions: 9038582 Slow queries: 310448 Opens: 28317 Open tables: 16119 Queries per second avg: 421.261 [07:30:47] we're up from 280,000 slow queries to 310,000, Reception123 :( [07:30:57] this is the time when we need John and Paladox :( [07:31:06] yeah.. [07:31:18] not sure why MariaDB finds that 'OK" [07:31:23] lol yes [07:31:45] or 421 queries per second seems quite high [07:32:59] db111 is having quite a nice day today [07:33:25] PROBLEM - db111 MariaDB on db111 is CRITICAL: Too many connections [07:34:44] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.2139434218 secs [07:37:25] RECOVERY - db111 MariaDB on db111 is OK: Uptime: 21932 Threads: 501 Questions: 9082808 Slow queries: 312014 Opens: 28568 Open tables: 16119 Queries per second avg: 414.134 [07:38:56] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.003075361252 secs [07:45:25] PROBLEM - db111 MariaDB on db111 is CRITICAL: Too many connections [07:46:08] PROBLEM - nonbinary.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/py [07:46:08] st-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query nonbinary.wiki. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:48:35] PROBLEM - miraheze.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/pyt [07:48:35] t-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query miraheze.wiki. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:49:25] RECOVERY - db111 MariaDB on db111 is OK: Uptime: 22652 Threads: 500 Questions: 9157665 Slow queries: 314234 Opens: 28900 Open tables: 16119 Queries per second avg: 404.276 [07:51:50] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.2031408846 secs [07:53:55] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.003049880266 secs [07:55:19] PROBLEM - robloxapi.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/py [07:55:19] st-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query robloxapi.wiki. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:58:18] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.3007747829 secs [07:59:28] PROBLEM - db111 MariaDB on db111 is CRITICAL: Too many connections [07:59:40] PROBLEM - bluearchive.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - bluearchive.wiki All nameservers failed to answer the query. [08:00:23] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.004852801561 secs [08:00:29] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [08:00:29] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.405271768569946 seconds [08:01:25] RECOVERY - db111 MariaDB on db111 is OK: Uptime: 23372 Threads: 500 Questions: 9215202 Slow queries: 315591 Opens: 29093 Open tables: 16119 Queries per second avg: 394.283 [08:04:46] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.2783388793 secs [08:06:51] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.00124758482 secs [08:09:25] PROBLEM - db111 MariaDB on db111 is CRITICAL: Too many connections [08:14:53] RECOVERY - nonbinary.wiki - reverse DNS on sslhost is OK: SSL OK - nonbinary.wiki reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK [08:17:08] RECOVERY - miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - miraheze.wiki reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK [08:17:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.79, 3.23, 2.17 [08:19:44] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.83, 2.82, 2.14 [08:24:38] RECOVERY - robloxapi.wiki - reverse DNS on sslhost is OK: SSL OK - robloxapi.wiki reverse DNS resolves to cp21.miraheze.org - CNAME FLAT [08:29:21] RECOVERY - bluearchive.wiki - reverse DNS on sslhost is OK: SSL OK - bluearchive.wiki reverse DNS resolves to cp21.miraheze.org - CNAME OK [08:30:25] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [08:32:28] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.203566879 secs [08:44:59] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.001531749964 secs [08:56:52] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[postgresql-contrib-13] [08:57:54] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.2043209672 secs [08:57:59] PROBLEM - matomo101 Disk Space on matomo101 is WARNING: DISK WARNING - free space: / 1035 MB (10% inode=80%); [09:00:00] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.001228034496 secs [09:24:51] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:29:04] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 8.086 second response time [09:32:57] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.2875556052 secs [09:33:16] PROBLEM - cp21 Stunnel HTTP for matomo101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:47:37] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.003324449062 secs [10:01:41] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.1973771155 secs [10:07:59] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.01468840241 secs [10:39:57] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.3807052672 secs [10:48:17] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.001428753138 secs [10:57:59] PROBLEM - db111 NTP time on db111 is WARNING: NTP WARNING: Offset -0.3283213973 secs [11:00:05] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset -0.003927946091 secs [11:08:28] [discord] Is anybody having this issue with logging in? I am, it seems. [11:08:29] [discord] https://cdn.discordapp.com/attachments/808001911868489748/947812495266033754/bandicam_2022-02-28_06-03-56-702.jpg [11:24:56] [discord] I'm having issues logging in, and I don't know if anybody else is having this issue. [11:30:29] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [11:30:29] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406675338745117 seconds [11:30:48] @DarkMatterBladeMan4000 yes, it is a database issue. Unfortunately I'm not sure how to resolve it and it will need John or Paladox [11:31:27] [discord] I had a feeling that might have been the case. [11:35:22] Oh @darkmatterman450 yeah [11:35:33] It's been really tough for me [11:35:51] I haven't been able to log in since morning [11:38:49] Glad I'm not the only one. [12:08:13] PROBLEM - db111 NTP time on db111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [12:08:29] PROBLEM - db111 SSH on db111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:47] PROBLEM - db111 Puppet on db111 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): Service[ulogd2],Service[prometheus-node-exporter],Service[ssh],Service[postfix] [12:08:53] RECOVERY - db111 APT on db111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:09:05] [02miraheze/GlobalNewFiles] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/dee52283b1b5...3dd015c5361f [12:09:07] [02miraheze/GlobalNewFiles] 07translatewiki 033dd015c - Localisation updates from https://translatewiki.net. [12:09:08] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/931a7c7511ae...b76684e76f80 [12:09:09] [02miraheze/CreateWiki] 07translatewiki 03b76684e - Localisation updates from https://translatewiki.net. [12:09:11] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/4bfa544cf4fb...a9f7e308acad [12:09:13] [02miraheze/landing] 07translatewiki 03a9f7e30 - Localisation updates from https://translatewiki.net. [12:09:14] [02miraheze/MatomoAnalytics] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MatomoAnalytics/compare/880ae2046731...f71aa73181a7 [12:09:16] [02miraheze/MatomoAnalytics] 07translatewiki 03f71aa73 - Localisation updates from https://translatewiki.net. [12:09:24] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 1.06, 0.33, 0.12 [12:10:15] RECOVERY - db111 NTP time on db111 is OK: NTP OK: Offset 0.0005064308643 secs [12:10:23] RECOVERY - db111 SSH on db111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [12:10:41] RECOVERY - db111 Puppet on db111 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [12:11:24] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 1.122 second response time [12:11:25] RECOVERY - db111 MariaDB on db111 is OK: Uptime: 167 Threads: 8 Questions: 13479 Slow queries: 973 Opens: 952 Open tables: 946 Queries per second avg: 80.712 [12:11:29] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.340 second response time [12:11:44] miraheze/landing - translatewiki the build passed. [12:11:53] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.589 second response time [12:11:56] RECOVERY - cp21 Stunnel HTTP for matomo101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.196 second response time [12:12:34] RECOVERY - cp30 Stunnel HTTP for matomo101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.553 second response time [12:17:49] miraheze/GlobalNewFiles - translatewiki the build passed. [12:19:08] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 9.74, 6.85, 3.35 [12:21:22] miraheze/CreateWiki - translatewiki the build passed. [12:23:44] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.16, 1.64, 1.02 [12:23:54] miraheze/MatomoAnalytics - translatewiki the build passed. [12:25:15] [discord] Well, I've managed to finally log in again. [12:25:44] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.53, 1.61, 1.08 [12:30:28] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [12:43:24] paladox: around? [12:43:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:45:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.50, 2.79, 2.23 [12:46:26] Only mobile [12:47:44] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.95, 2.37, 2.14 [12:48:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.50, 8.73, 6.86 [12:50:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.76, 8.71, 7.11 [12:52:05] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.58, 7.89, 7.00 [12:53:44] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.30, 2.06, 1.55 [12:55:44] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.39, 1.77, 1.50 [12:56:29] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.61, 6.59, 5.83 [12:57:44] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.22, 1.61, 1.48 [12:58:27] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.61, 5.89, 5.68 [13:03:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:10:45] PROBLEM - db111 APT on db111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [13:17:12] RECOVERY - db111 APT on db111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:26:06] PROBLEM - db111 APT on db111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [13:30:29] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [13:30:29] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.405102968215942 seconds [13:34:04] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.78, 1.76, 1.33 [13:36:03] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.48, 1.32, 1.22 [13:37:59] PROBLEM - matomo101 Disk Space on matomo101 is CRITICAL: DISK CRITICAL - free space: / 565 MB (5% inode=80%); [13:40:40] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:45:40] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:54:06] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.93, 1.53, 1.18 [13:54:21] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.30, 5.76, 5.10 [13:54:40] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:56:19] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.05, 5.59, 5.13 [13:58:06] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.95, 1.38, 1.22 [14:00:26] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [14:00:37] what I get now is `The provided authentication token is either expired or invalid.` [14:00:59] and I am logged in on loginwiki but not centrally logged in [14:04:40] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:09:53] okay, I'm in now [14:15:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:19:44] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.63, 8.46, 7.60 [14:21:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.36, 8.36, 7.67 [14:40:12] PROBLEM - wiki.bikinibottomdiscord.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.bikinibottomdiscord.org All nameservers failed to answer the query. [14:43:16] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.40, 8.24, 6.96 [14:45:12] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.54, 7.53, 6.85 [14:45:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:53:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:56:25] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [14:56:25] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:01:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.24, 2.61 [15:03:09] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.42, 2.77, 2.50 [15:08:39] ok : [RESOLVED] (yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:09:23] RECOVERY - wiki.bikinibottomdiscord.org - reverse DNS on sslhost is OK: SSL OK - wiki.bikinibottomdiscord.org reverse DNS resolves to cp20.miraheze.org - CNAME OK [15:12:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.45, 8.26, 7.62 [15:13:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/30966717a668...baa57f8cd1c5 [15:13:29] [02miraheze/puppet] 07paladox 03baa57f8 - elasticsearch: Allow grafana access to port 9206 [15:14:17] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.14, 7.61, 7.47 [15:21:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/baa57f8cd1c5...34ec070ccf0c [15:21:58] [02miraheze/puppet] 07paladox 0334ec070 - grafana: Set proxy_set_header to 'Host $http_host' for / [15:23:41] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.22, 1.77, 1.16 [15:25:20] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [15:25:41] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.20, 1.57, 1.16 [15:30:29] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [15:30:29] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.405405282974243 seconds [15:37:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:40:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/af1bc6b1249f [15:40:37] [02miraheze/puppet] 07paladox 03af1bc6b - varnish: Temporarily block access for Wiki-Bot due to instability with the dns [15:40:39] [02puppet] 07paladox created branch 03paladox-patch-7 - 13https://github.com/miraheze/puppet [15:40:40] [02puppet] 07paladox opened pull request 03#2439: varnish: Temporarily block access for Wiki-Bot due to instability wit… - 13https://github.com/miraheze/puppet/pull/2439 [15:40:51] [02puppet] 07paladox edited pull request 03#2439: varnish: Temporarily block access for Wiki-Bot due to instability with the dbs - 13https://github.com/miraheze/puppet/pull/2439 [15:40:57] [02puppet] 07paladox edited pull request 03#2439: varnish: Temporarily block access for Wiki-Bot due to instability with the dbs - 13https://github.com/miraheze/puppet/pull/2439 [15:44:57] [02puppet] 07paladox closed pull request 03#2439: varnish: Temporarily block access for Wiki-Bot due to instability with the dbs - 13https://github.com/miraheze/puppet/pull/2439 [15:44:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/34ec070ccf0c...0b279125de9c [15:45:00] [02miraheze/puppet] 07paladox 030b27912 - varnish: Temporarily block access for Wiki-Bot due to instability with the dns (#2439) [15:45:01] [02puppet] 07paladox deleted branch 03paladox-patch-7 - 13https://github.com/miraheze/puppet [15:45:03] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-7 [15:51:14] !log restart mysql on db111 [15:51:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:52:02] paladox: are you able to deploy the MariaDB config change I requested on task [15:52:02] [tell] RhinosF1: 2022-02-28 - 03:02:29GMT tell RhinosF1 following the recent events, I think naleksuh needs to be banned from bots. [15:52:21] Or would you prefer I attempt to backport the mediawiki patch [15:52:27] what change [15:52:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:52:45] paladox: adding a limit to query execution time so they can't go on for 10 minutes [15:52:53] Which I understand was happening [15:52:56] MacFan4000: do it [15:52:58] RECOVERY - db111 APT on db111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:52:59] it should already have a limit [15:53:07] PROBLEM - cp20 Stunnel HTTP for matomo101 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2415 bytes in 0.103 second response time [15:53:15] paladox: apparently not a working one from what void was saying [15:53:15] PROBLEM - cp31 Stunnel HTTP for matomo101 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2415 bytes in 0.311 second response time [15:53:18] PROBLEM - matomo101 HTTPS on matomo101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2415 bytes in 0.047 second response time [15:53:25] PROBLEM - db111 MariaDB on db111 is CRITICAL: Can't connect to MySQL server on 'db111.miraheze.org' (115) [15:53:46] https://phabricator.miraheze.org/T8851#178991 [15:53:46] [url] ⚓ T8851 db111 insanely high load | phabricator.miraheze.org [15:55:04] RECOVERY - cp20 Stunnel HTTP for matomo101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.229 second response time [15:55:13] RECOVERY - cp31 Stunnel HTTP for matomo101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 66463 bytes in 0.589 second response time [15:55:18] RECOVERY - matomo101 HTTPS on matomo101 is OK: HTTP OK: HTTP/1.1 200 OK - 66479 bytes in 0.332 second response time [15:55:25] RECOVERY - db111 MariaDB on db111 is OK: Uptime: 128 Threads: 13 Questions: 48731 Slow queries: 1725 Opens: 2347 Open tables: 2341 Queries per second avg: 380.710 [15:55:42] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/747142/ won't really fix anything [15:55:48] we have a timeout nginx/php side [15:55:54] it's mysql won't timeout [15:57:03] uh how is it still managing to do: [15:57:20] | 286 | mediawiki | 2a10:6740::6:309:48050 | mogaiwiki | Query | 171 | Sending data | SELECT /* ApiQueryRecentChanges::run */ actor_name,actor_user,rc_actor,rc_id,rc_timestamp,rc_namespace,rc_title,rc_cur_id,rc_type,rc_deleted,rc_this_oldid,rc_last_oldid,rc_minor,rc_type,rc_bot,rc_old_len,rc_new_len,rc_logid,rc_log_type,rc_log_action,rc_params,(SELECT GROUP_CONCAT(ctd_name SEPARATOR ',') FROM `change_tag` JOIN `change_tag_def` [15:57:20] ON ((ct_tag_id=ctd_id)) WHERE ct_rc_id=rc_id ) AS `ts_tags`,page_is_redirect,comment_rc_comment.comment_text AS `rc_comment_text`,comment_rc_comment.comment_data AS `rc_comment_data`,comment_rc_comment.comment_id AS `rc_comment_cid` FROM `recentchanges` JOIN `actor` ON ((actor_id=rc_actor)) LEFT JOIN `page` ON ((rc_namespace=page_namespace) AND (rc_title=page_title)) JOIN `comment` `comment_rc_comment` ON [15:57:20] ((comment_rc_comment.comment_id = rc_comment_id)) WHERE rc_type IN (0,1,3,6) AND (rc_bot = 0) ORDER BY rc_timestamp DESC,rc_id DESC LIMIT 21 | 0.000 | [15:57:52] paladox: why aren't they being killed [15:58:11] Maybe too much load for query killer? [15:58:40] only timeouts we have are https://github.com/miraheze/puppet/blob/master/modules/mariadb/templates/config/mw.cnf.erb#L57 [15:58:41] [url] puppet/mw.cnf.erb at master · miraheze/puppet · GitHub | github.com [16:01:16] paladox: can we please set max_execution_time [16:01:22] To something sensible [16:01:37] As the queries clearly aren't dying after php/nginx does [16:01:50] either for only expensive pages via that patch or globally [16:04:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:05:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.94, 8.51, 7.74 [16:05:46] doesn't seem like max_execution_time exists in MariaDB RhinosF1 [16:05:51] mariadb has https://mariadb.com/kb/en/aborting-statements/ though [16:05:53] [url] Aborting Statements that Exceed a Certain Time to Execute - MariaDB Knowledge Base | mariadb.com [16:06:09] paladox: I bribed reedy into making a backport too [16:06:13] it should exist [16:06:25] Ok, but that change doesn't do much [16:06:37] Wikimedia use a query killer anyways [16:06:55] RhinosF1: nope, https://jira.mariadb.org/browse/MDEV-9829 [16:06:56] [url] [MDEV-9829] Merge max_execution_time from MySQL 5.7 - Jira | jira.mariadb.org [16:07:19] paladox: it will stop the recent changes queries over running [16:07:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.73, 8.42, 7.82 [16:07:36] how, we have timeouts [16:07:38] You can try max_statement_time [16:07:39] in php/nginx [16:07:44] it basically just does the same thing [16:07:50] paladox: but they obviously aren't killing the query [16:08:03] yes so that change won't either [16:08:04] And this would be way lower [16:08:24] ohhh [16:08:26] i misread it [16:08:29] All that does is tell MariaDB do not let this query take longer than x milliseconds [16:09:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:10:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.76, 8.50, 8.03 [16:11:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.51, 8.52, 7.98 [16:12:05] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.42, 8.50, 8.08 [16:12:50] RhinosF1: i think maybe best to get that backport deployed [16:13:08] paladox: I am bribing reedy as we speak [16:13:19] Safer to have it in upstream too [16:13:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.57, 3.53, 2.75 [16:13:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:13:53] ok [16:14:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 3.68, 3.06 [16:15:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.41, 8.45, 8.12 [16:15:35] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.29, 3.52, 2.85 [16:16:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.35, 8.82, 8.38 [16:16:17] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.36, 3.14, 2.93 [16:17:35] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.38, 3.13, 2.79 [16:18:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.83, 9.58, 8.71 [16:19:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.64, 9.02, 8.44 [16:21:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.09, 8.48, 8.33 [16:22:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.85, 9.49, 8.89 [16:26:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.15, 9.31, 8.92 [16:27:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.60, 8.52, 8.39 [16:28:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.34, 9.36, 8.99 [16:29:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.93, 8.32, 8.33 [16:32:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.58, 9.50, 9.09 [16:33:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.28, 9.56, 8.90 [16:34:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.46, 8.86, 8.92 [16:34:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/4fc1ae7b71f6 [16:34:45] [02miraheze/puppet] 07paladox 034fc1ae7 - icinga2: Add proxy for apt [16:34:47] [02puppet] 07paladox created branch 03paladox-patch-7 - 13https://github.com/miraheze/puppet [16:34:48] [02puppet] 07paladox opened pull request 03#2440: icinga2: Add proxy for apt - 13https://github.com/miraheze/puppet/pull/2440 [16:36:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4fc1ae7b71f6...bb0f07bc00a6 [16:36:09] [02miraheze/puppet] 07paladox 03bb0f07b - Update 01gluster.erb [16:36:10] [02puppet] 07paladox synchronize pull request 03#2440: icinga2: Add proxy for apt - 13https://github.com/miraheze/puppet/pull/2440 [16:36:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bb0f07bc00a6...b5745c60fe85 [16:36:23] [02miraheze/puppet] 07paladox 03b5745c6 - Update mon111.yaml [16:36:25] [02puppet] 07paladox synchronize pull request 03#2440: icinga2: Add proxy for apt - 13https://github.com/miraheze/puppet/pull/2440 [16:36:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/b5745c60fe85...ca58b9caeb81 [16:36:56] [02miraheze/puppet] 07paladox 03ca58b9c - Create 01icinga.erb [16:36:57] [02puppet] 07paladox synchronize pull request 03#2440: icinga2: Add proxy for apt - 13https://github.com/miraheze/puppet/pull/2440 [16:37:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ca58b9caeb81...8ffc273663dc [16:37:12] [02miraheze/puppet] 07paladox 038ffc273 - Update 01gluster.erb [16:37:13] [02puppet] 07paladox synchronize pull request 03#2440: icinga2: Add proxy for apt - 13https://github.com/miraheze/puppet/pull/2440 [16:40:37] [02puppet] 07paladox closed pull request 03#2440: icinga2: Add proxy for apt - 13https://github.com/miraheze/puppet/pull/2440 [16:40:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/miraheze/puppet/compare/0b279125de9c...fe97ef53de4a [16:40:40] [02miraheze/puppet] 07paladox 03fe97ef5 - icinga2: Add proxy for apt (#2440) [16:40:41] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-7 [16:40:43] [02puppet] 07paladox deleted branch 03paladox-patch-7 - 13https://github.com/miraheze/puppet [16:42:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.75, 8.91, 8.78 [16:42:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.83, 8.31, 7.57 [16:43:00] [02miraheze/mediawiki] 07RhinosF1 pushed 0323 commits to 03REL1_37 [+3/-0/±192] 13https://github.com/miraheze/mediawiki/compare/1049560a92dd...4093cfbf4d1e [16:43:01] [02miraheze/mediawiki] 07translatewiki 0379c1fea - Localisation updates from https://translatewiki.net. [16:43:03] [02miraheze/mediawiki] 07translatewiki 039729d66 - Update git submodules [16:43:04] [02miraheze/mediawiki] 07translatewiki 03651f395 - Update git submodules [16:43:06] [02miraheze/mediawiki] ... and 20 more commits. [16:43:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.28, 3.71, 3.05 [16:43:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.52, 9.25, 8.83 [16:44:47] !log [rhinos@mwtask111] starting deploy of {'world': True, 'l10n': True} to all [16:44:49] !log [rhinos@test101] starting deploy of {'world': True, 'l10n': True} to all [16:44:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:44:59] paladox: fyi ^ [16:45:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:45:10] RhinosF1: thanks! [16:45:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.27, 9.26, 8.89 [16:46:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.14, 9.19, 8.95 [16:46:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.12, 3.45, 2.83 [16:47:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.84, 3.28 [16:47:59] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/9a75eab06881 [16:48:00] [02miraheze/mw-config] 07RhinosF1 039a75eab - Database: limit db query for expensive pages to 20s [16:48:02] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-3 - 13https://github.com/miraheze/mw-config [16:48:04] [02mw-config] 07RhinosF1 opened pull request 03#4485: Database: limit db query for expensive pages to 20s - 13https://github.com/miraheze/mw-config/pull/4485 [16:48:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.53, 3.05, 2.76 [16:49:11] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.03, 4.06, 3.41 [16:49:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.82, 9.97, 9.26 [16:49:54] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9a75eab06881...2f28bd7413fe [16:49:56] [02miraheze/mw-config] 07github-actions 032f28bd7 - CI: lint code to MediaWiki standards [16:49:57] [02mw-config] 07github-actions[bot] synchronize pull request 03#4485: Database: limit db query for expensive pages to 20s - 13https://github.com/miraheze/mw-config/pull/4485 [16:50:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.19, 9.92, 9.27 [16:50:13] !log restart redis-server on matomo101 [16:50:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:50:35] miraheze/mw-config - RhinosF1 the build passed. [16:50:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.44, 8.49, 8.05 [16:53:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.74, 3.44, 3.30 [16:53:39] [02CreateWiki] 07Universal-Omega closed pull request 03#305: Fix WikiManager::delete - 13https://github.com/miraheze/CreateWiki/pull/305 [16:53:41] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/b76684e76f80...35542b34cf30 [16:53:42] [02miraheze/CreateWiki] 07Universal-Omega 0335542b3 - Fix WikiManager::delete (#305) [16:53:44] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.99, 1.51, 0.93 [16:53:44] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/CreateWiki [16:53:45] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [16:53:59] RECOVERY - matomo101 Disk Space on matomo101 is OK: DISK OK - free space: / 1374 MB (14% inode=80%); [16:54:59] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.81, 3.09, 3.18 [16:55:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.09, 9.50, 9.39 [16:55:44] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.62, 1.57, 1.02 [16:56:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.39, 9.74, 9.47 [16:57:07] PROBLEM - db111 APT on db111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [16:57:15] !log restart mysql on db111 [16:57:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:57:23] RhinosF1: Reception123 ^ [16:57:32] should kill all the queries. [16:57:42] let's hope so [16:58:54] miraheze/CreateWiki - Universal-Omega the build passed. [16:59:05] RECOVERY - db111 APT on db111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:59:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.58, 4.20, 3.37 [16:59:30] !log [rhinos@test101] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 880s [16:59:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:00:28] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [17:01:11] the dns servers for that domain dont seem to be very stable [17:01:17] theres always alerts for it [17:01:36] MacFan4000: can you tell why [17:03:14] not sure why [17:03:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.39, 3.43, 3.26 [17:03:53] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/bcbcd1c94c65...97335c40bb7c [17:03:55] [02miraheze/CreateWiki] 07Universal-Omega 0397335c4 - Update RemoteWikiTest.php [17:03:56] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [17:04:05] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.83, 7.60, 8.49 [17:05:18] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.61, 3.18, 3.18 [17:06:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/61d4007dad6a [17:06:43] [02miraheze/puppet] 07paladox 0361d4007 - redis: Fix redis systemd config [17:06:44] [02puppet] 07paladox created branch 03paladox-patch-7 - 13https://github.com/miraheze/puppet [17:06:45] [02puppet] 07paladox opened pull request 03#2441: redis: Fix redis systemd config - 13https://github.com/miraheze/puppet/pull/2441 [17:07:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/61d4007dad6a...8dfee72e18e0 [17:07:03] [02miraheze/puppet] 07paladox 038dfee72 - Update redis-server.systemd.erb [17:07:05] [02puppet] 07paladox synchronize pull request 03#2441: redis: Fix redis systemd config - 13https://github.com/miraheze/puppet/pull/2441 [17:07:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8dfee72e18e0...e9e579632712 [17:07:21] [02miraheze/puppet] 07paladox 03e9e5796 - Update redis-server.systemd.erb [17:07:23] [02puppet] 07paladox synchronize pull request 03#2441: redis: Fix redis systemd config - 13https://github.com/miraheze/puppet/pull/2441 [17:07:48] [02puppet] 07paladox closed pull request 03#2441: redis: Fix redis systemd config - 13https://github.com/miraheze/puppet/pull/2441 [17:07:50] [02puppet] 07paladox deleted branch 03paladox-patch-7 - 13https://github.com/miraheze/puppet [17:07:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fe97ef53de4a...5f4cb32926da [17:07:53] [02miraheze/puppet] 07paladox 035f4cb32 - redis: Fix redis systemd config (#2441) [17:07:54] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-7 [17:08:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.98, 8.56, 8.70 [17:08:38] miraheze/CreateWiki - Universal-Omega the build passed. [17:09:44] !log [rhinos@mwtask111] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 1497s [17:09:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:10:19] RECOVERY - matomo101 Redis Process on matomo101 is OK: PROCS OK: 1 process with args 'redis-server' [17:10:28] [02mw-config] 07RhinosF1 closed pull request 03#4485: Database: limit db query for expensive pages to 20s - 13https://github.com/miraheze/mw-config/pull/4485 [17:10:30] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/1a320b0750ac...1da729bbc592 [17:10:31] [02miraheze/mw-config] 07RhinosF1 031da729b - Database: limit db query for expensive pages to 20s (#4485) [17:10:33] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-3 [17:10:34] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-3 - 13https://github.com/miraheze/mw-config [17:11:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.65, 3.26, 3.17 [17:11:17] PROBLEM - matomo101 Puppet on matomo101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[redis-server] [17:11:35] miraheze/mw-config - RhinosF1 the build passed. [17:13:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.68, 2.65, 2.96 [17:13:16] RECOVERY - matomo101 Puppet on matomo101 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [17:13:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.14, 2.83 [17:15:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.64, 2.87, 2.76 [17:18:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.97, 9.92, 9.18 [17:18:10] !log [@test101] starting deploy of {'config': True} to all [17:18:11] !log [@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [17:18:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:18:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:20:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.41, 9.45, 9.12 [17:20:34] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [17:21:19] !log [rhinos@test101] starting deploy of {'l10n': True} to all [17:21:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.35, 9.33, 9.00 [17:21:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:22:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.95, 10.69, 9.63 [17:22:49] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1da729bbc592...62f0c32e2214 [17:22:50] [02miraheze/mw-config] 07RhinosF1 0362f0c32 - Update Database.php [17:23:01] !log [rhinos@test101] starting deploy of {'pull': 'config', 'config': True} to all [17:23:02] !log [rhinos@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [17:23:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:23:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:23:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.92, 9.08, 8.95 [17:23:59] miraheze/mw-config - RhinosF1 the build passed. [17:24:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.01, 9.95, 9.49 [17:25:04] !log [@mwtask111] starting deploy of {'config': True} to all [17:25:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:25:29] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 24s [17:25:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:25:50] !log [rhinos@mwtask111] enabled puppet [17:26:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:26:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.14, 9.99, 9.55 [17:27:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.85, 9.21, 8.97 [17:27:54] !log restart mysql on db111 [17:27:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:29:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.41, 8.81, 8.86 [17:30:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 5.91, 8.53, 9.14 [17:30:29] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [17:30:29] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404927730560303 seconds [17:33:08] [discord] Load is looking rather more promising [17:33:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:34:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.80, 9.17, 9.19 [17:35:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.94, 9.71, 9.08 [17:37:00] [discord] @Orduin paladox says the patch isn't working [17:37:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.16, 9.02, 8.90 [17:37:25] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/e1dc93a560b9 [17:37:26] [02miraheze/ssl] 07MacFan4000 03e1dc93a - Delete wiki.landev.vn.crt [17:37:28] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [17:37:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:38:07] [02ssl] 07MacFan4000 opened pull request 03#484: Remove wiki.landev.vn and www.plan.eu.org - 13https://github.com/miraheze/ssl/pull/484 [17:38:48] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/e1dc93a560b9...3ebb15d01cf8 [17:38:50] [02miraheze/ssl] 07MacFan4000 033ebb15d - Delete www.plan.eu.org.crt [17:38:52] [02ssl] 07MacFan4000 synchronize pull request 03#484: Remove wiki.landev.vn and www.plan.eu.org - 13https://github.com/miraheze/ssl/pull/484 [17:40:02] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3ebb15d01cf8...5030c99cae4d [17:40:03] [02miraheze/ssl] 07MacFan4000 035030c99 - Update certs.yaml [17:40:05] [02ssl] 07MacFan4000 synchronize pull request 03#484: Remove wiki.landev.vn and www.plan.eu.org - 13https://github.com/miraheze/ssl/pull/484 [17:40:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.93, 9.26, 9.31 [17:45:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.72, 7.97, 8.46 [17:47:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:48:05] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.32, 7.73, 8.50 [17:48:39] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [17:52:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.34, 8.98, 8.83 [17:52:51] [02ssl] 07MacFan4000 closed pull request 03#484: Remove wiki.landev.vn and www.plan.eu.org - 13https://github.com/miraheze/ssl/pull/484 [17:52:53] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-2/±1] 13https://github.com/miraheze/ssl/compare/df9d2c29c27b...ed3810d05713 [17:52:54] [02miraheze/ssl] 07MacFan4000 03ed3810d - Remove wiki.landev.vn and www.plan.eu.org (#484) [17:52:56] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [17:52:57] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [17:54:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.06, 8.69, 8.73 [17:56:05] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.04, 9.99, 9.21 [17:56:25] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [17:57:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.65, 9.86, 8.92 [17:57:47] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[wiki.landev.vn_private],File[www.plan.eu.org_private] [17:59:02] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.86, 8.35, 7.79 [17:59:08] both private and public keys are removed, so i suspects thats temporary (puppet failures) [17:59:09] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[wiki.landev.vn_private],File[www.plan.eu.org_private] [17:59:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.32, 9.55, 8.92 [18:00:25] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [18:00:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:01:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.15, 8.79, 8.01 [18:02:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.88, 9.42, 9.29 [18:02:59] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.37, 8.42, 7.98 [18:05:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.26, 9.90, 9.24 [18:05:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/f1d0d504c922 [18:05:43] [02miraheze/puppet] 07paladox 03f1d0d50 - varnish: Fix blocking Wiki-Bot UA [18:05:44] [02puppet] 07paladox created branch 03paladox-patch-7 - 13https://github.com/miraheze/puppet [18:05:46] [02puppet] 07paladox opened pull request 03#2442: varnish: Fix blocking Wiki-Bot UA - 13https://github.com/miraheze/puppet/pull/2442 [18:06:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.74, 8.40, 7.55 [18:06:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f1d0d504c922...ccd1c2b43f53 [18:06:54] [02miraheze/puppet] 07paladox 03ccd1c2b - Update mediawiki.conf [18:06:55] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.89, 8.92, 8.27 [18:06:55] [02puppet] 07paladox synchronize pull request 03#2442: varnish: Fix blocking Wiki-Bot UA - 13https://github.com/miraheze/puppet/pull/2442 [18:07:02] [02puppet] 07paladox closed pull request 03#2442: varnish: Fix blocking Wiki-Bot UA - 13https://github.com/miraheze/puppet/pull/2442 [18:07:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5f4cb32926da...5ea183da88f5 [18:07:05] [02miraheze/puppet] 07paladox 035ea183d - varnish: Fix blocking Wiki-Bot UA (#2442) [18:07:07] [02puppet] 07paladox deleted branch 03paladox-patch-7 - 13https://github.com/miraheze/puppet [18:07:08] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-7 [18:07:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.62, 8.95, 8.97 [18:08:53] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.21, 8.07, 8.03 [18:09:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.39, 9.40, 9.12 [18:09:45] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [18:10:36] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.36, 9.15, 8.05 [18:10:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.02, 9.35, 7.93 [18:11:07] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:11:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.89, 3.82, 3.09 [18:11:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.95, 9.24, 9.10 [18:12:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.65, 9.20, 8.23 [18:12:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.89, 9.71, 8.25 [18:13:44] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.95, 1.46, 0.93 [18:13:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.63, 4.25, 3.33 [18:15:07] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.10, 3.60, 3.16 [18:15:08] [02puppet] 07Universal-Omega commented on pull request 03#2423: Allow static access from same domain - 13https://github.com/miraheze/puppet/pull/2423#issuecomment-1054530962 [18:15:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:15:44] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.44, 1.48, 1.01 [18:15:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.40, 3.60, 3.20 [18:16:05] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.11, 7.69, 8.43 [18:16:39] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 0.96, 2.41, 7.97 [18:17:07] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.87, 2.96, 2.98 [18:17:44] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.27, 3.13, 3.08 [18:18:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.82, 8.13, 8.14 [18:20:39] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 1.07, 1.58, 6.35 [18:21:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:22:05] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.76, 9.09, 8.84 [18:22:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.70, 8.50, 8.14 [18:23:44] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.85, 1.69, 1.27 [18:24:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.80, 8.24, 8.30 [18:24:40] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.79, 7.92, 7.97 [18:25:20] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [18:25:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.87, 8.02, 8.45 [18:25:44] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.63, 1.65, 1.30 [18:26:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:31:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.02, 7.75, 8.26 [18:35:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.72, 8.83, 8.59 [18:35:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:40:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:43:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:45:08] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.57, 8.24, 8.45 [18:48:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:48:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.69, 9.18, 8.75 [18:54:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:54:42] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.91, 7.39, 8.13 [18:56:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.39, 8.61, 8.36 [18:57:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.50, 2.89, 2.38 [18:58:32] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.19, 9.84, 8.90 [18:59:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.89, 2.48, 2.29 [18:59:51] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [18:59:51] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406055927276611 seconds [19:00:00] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.47, 8.29, 8.29 [19:00:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.45, 8.85, 8.13 [19:01:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.14, 8.52, 7.85 [19:02:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.26, 9.17, 8.36 [19:03:36] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.99, 8.34, 7.87 [19:04:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.06, 9.73, 8.65 [19:06:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.75, 9.55, 8.73 [19:08:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.07, 9.71, 8.88 [19:10:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.10, 9.07, 8.74 [19:10:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.27, 9.66, 9.60 [19:10:34] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.01, 1.53, 1.20 [19:10:45] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.52, 9.18, 8.54 [19:12:32] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.70, 1.19, 1.11 [19:12:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.67, 8.84, 8.49 [19:15:19] [discord] We have two broken queries running on db111 that appear to be related to Special:Contributions, I'm on my way back now to see if I can look at it [19:16:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.63, 10.33, 9.83 [19:16:38] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.45, 8.09, 8.27 [19:17:02] @Orduin: we are discussing on irc [19:17:32] Lead suspect is a dodgy query plan [19:18:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.90, 9.99, 9.76 [19:19:03] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/97335c40bb7c...f469cfdcf048 [19:19:04] [02miraheze/CreateWiki] 07Universal-Omega 03f469cfd - Update RemoteWikiTest.php [19:19:06] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [19:19:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:20:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.29, 10.94, 10.14 [19:20:51] [discord] @Orduin [19:22:41] miraheze/CreateWiki - Universal-Omega the build has errored. [19:23:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:24:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.40, 9.75, 9.28 [19:24:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.54, 8.95, 9.54 [19:26:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.57, 9.73, 9.34 [19:28:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.64, 8.53, 8.06 [19:29:26] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [19:30:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.02, 9.81, 9.41 [19:32:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.66, 9.36, 9.30 [19:34:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.24, 9.76, 9.46 [19:34:50] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.86, 9.65, 8.70 [19:37:54] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [19:37:56] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/f469cfdcf048...8628ef5e7fad [19:37:57] [02miraheze/CreateWiki] 07Universal-Omega 038628ef5 - Update RemoteWikiTest.php [19:38:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.27, 9.56, 8.95 [19:40:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.84, 9.86, 9.67 [19:44:11] miraheze/CreateWiki - Universal-Omega the build has errored. [19:45:21] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/8628ef5e7fad...bfb9d7aa07ac [19:45:23] [02miraheze/CreateWiki] 07Universal-Omega 03bfb9d7a - Update RemoteWikiTest.php [19:45:24] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [19:48:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.89, 9.31, 9.36 [19:48:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.61, 9.66, 9.41 [19:48:30] miraheze/CreateWiki - Universal-Omega the build has errored. [19:49:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.84, 8.61, 8.28 [19:49:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.09, 8.86, 8.11 [19:49:13] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/bfb9d7aa07ac...588649e53590 [19:49:15] [02miraheze/CreateWiki] 07Universal-Omega 03588649e - Update RemoteWikiTest.php [19:49:16] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [19:51:05] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.67, 8.50, 8.07 [19:52:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.03, 9.37, 9.44 [19:52:40] miraheze/CreateWiki - Universal-Omega the build has errored. [19:53:04] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.16, 8.48, 8.30 [19:54:38] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.87, 1.24, 1.03 [19:54:52] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/588649e53590...8e2922dd1207 [19:54:54] [02miraheze/CreateWiki] 07Universal-Omega 038e2922d - Update RemoteWikiTest.php [19:54:55] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [19:55:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.09, 8.75, 8.27 [19:56:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.05, 1.23, 1.06 [19:57:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.70, 8.86, 8.51 [19:58:19] miraheze/CreateWiki - Universal-Omega the build has errored. [19:58:50] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.33, 8.79, 8.71 [19:59:03] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [19:59:03] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.403452157974243 seconds [19:59:04] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.06, 8.13, 8.31 [20:00:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.97, 9.34, 9.87 [20:00:49] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/8e2922dd1207...96289aa70115 [20:00:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.40, 9.00, 8.80 [20:00:51] [02miraheze/CreateWiki] 07Universal-Omega 0396289aa - Update RemoteWikiTest.php [20:00:52] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [20:00:59] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.00, 8.49, 8.38 [20:02:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.19, 7.11, 8.28 [20:03:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:03:58] miraheze/CreateWiki - Universal-Omega the build has errored. [20:06:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.68, 7.49, 8.24 [20:07:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:12:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.42, 7.24, 8.38 [20:13:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.10, 8.21, 8.11 [20:15:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.74, 8.42, 8.32 [20:15:49] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.36, 7.95, 8.00 [20:17:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.97, 8.61, 7.93 [20:18:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.05, 8.53, 8.60 [20:18:43] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/96289aa70115...4d7d6b829253 [20:18:45] [02miraheze/CreateWiki] 07Universal-Omega 034d7d6b8 - Update RemoteWikiTest.php [20:18:46] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [20:19:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.69, 7.78, 7.71 [20:20:16] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.60, 8.76, 8.50 [20:20:19] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/4d7d6b829253...6ba5c87cf701 [20:20:20] [02miraheze/CreateWiki] 07Universal-Omega 036ba5c87 - Update RemoteWikiTest.php [20:20:22] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [20:21:52] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/6ba5c87cf701...842780e1b4b3 [20:21:53] [02miraheze/CreateWiki] 07Universal-Omega 03842780e - Update RemoteWikiTest.php [20:21:55] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [20:22:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.18, 3.43, 3.11 [20:22:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.56, 8.70, 8.33 [20:23:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.06, 7.85, 8.22 [20:23:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.99, 3.02, 3.00 [20:24:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.54, 8.22, 8.34 [20:24:40] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.09, 7.81, 8.06 [20:26:00] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [20:26:00] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:28:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.05, 8.93, 8.80 [20:28:25] miraheze/CreateWiki - Universal-Omega the build passed. [20:32:00] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/842780e1b4b3...5abd65b35495 [20:32:02] [02miraheze/CreateWiki] 07Universal-Omega 035abd65b - Update RemoteWikiTest.php [20:32:03] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [20:32:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.06, 8.59, 8.73 [20:32:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:36:14] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/5abd65b35495...e3f5838c8d2a [20:36:16] [02miraheze/CreateWiki] 07Universal-Omega 03e3f5838 - Update RemoteWikiTest.php [20:36:17] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [20:36:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.68, 7.72, 8.36 [20:42:04] miraheze/CreateWiki - Universal-Omega the build passed. [20:42:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.71, 8.52, 8.13 [20:42:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.98, 9.65, 8.94 [20:42:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:44:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.56, 8.92, 8.76 [20:46:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.69, 9.68, 9.06 [20:51:09] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/e3f5838c8d2a...f5d31a9cedef [20:51:11] [02miraheze/CreateWiki] 07Universal-Omega 03f5d31a9 - Update RemoteWikiTest.php [20:51:12] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [20:51:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.09, 8.71, 8.12 [20:52:01] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.08, 9.31, 8.68 [20:53:53] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.02, 9.28, 8.40 [20:53:59] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.21, 9.10, 8.67 [20:55:51] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.36, 9.20, 8.50 [20:55:57] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.36, 9.99, 9.05 [20:57:22] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.22, 9.24, 8.32 [20:58:14] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [20:58:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.06, 9.33, 9.72 [20:59:08] miraheze/CreateWiki - Universal-Omega the build passed. [20:59:17] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.58, 8.79, 8.25 [20:59:53] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.63, 9.50, 9.07 [21:01:13] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.63, 9.53, 8.56 [21:02:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.41, 9.74, 9.80 [21:03:08] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.53, 8.91, 8.47 [21:03:43] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.57, 8.23, 8.30 [21:04:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.80, 8.77, 9.45 [21:04:30] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/f5d31a9cedef...b936618ef84c [21:04:32] [02miraheze/CreateWiki] 07Universal-Omega 03b936618 - Update RemoteWikiTest.php [21:04:33] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [21:06:59] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.46, 8.05, 8.21 [21:07:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:09:34] miraheze/CreateWiki - Universal-Omega the build has errored. [21:09:42] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.39, 8.99, 8.90 [21:10:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.77, 9.29, 9.36 [21:10:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.79, 8.77, 8.55 [21:10:52] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.98, 9.00, 8.55 [21:11:40] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.36, 8.96, 8.91 [21:12:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.89, 9.07, 9.27 [21:12:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.31, 8.44, 8.47 [21:12:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:12:50] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.12, 9.63, 8.83 [21:13:09] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/b936618ef84c...ece0ca44b68e [21:13:10] [02miraheze/CreateWiki] 07Universal-Omega 03ece0ca4 - Update RemoteWikiTest.php [21:13:12] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [21:13:38] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.07, 9.62, 9.15 [21:14:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.05, 9.38, 9.35 [21:16:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.08, 9.25, 9.33 [21:16:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.34, 8.58, 8.53 [21:16:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.63, 3.48, 2.89 [21:16:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.69, 9.52, 8.99 [21:17:03] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/ece0ca44b68e...457209d44211 [21:17:05] [02miraheze/CreateWiki] 07Universal-Omega 03457209d - Update RemoteWikiTest.php [21:17:06] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [21:18:04] miraheze/CreateWiki - Universal-Omega the build has errored. [21:18:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.21, 9.60, 9.43 [21:18:23] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.18, 8.48, 8.50 [21:20:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.85, 9.87, 9.56 [21:20:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.35, 3.15, 2.91 [21:21:30] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.18, 9.17, 9.33 [21:22:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.30, 10.54, 9.83 [21:23:19] miraheze/CreateWiki - Universal-Omega the build has errored. [21:26:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.68, 9.49, 9.62 [21:27:50] PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 123, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 108, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn [21:27:51] AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407571077346802 seconds [21:28:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.49, 10.29, 9.92 [21:28:50] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.73, 8.91, 8.83 [21:29:22] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.51, 8.95, 9.02 [21:29:46] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/457209d44211...f79d47cab7ea [21:29:48] [02miraheze/CreateWiki] 07Universal-Omega 03f79d47c - Update RemoteWikiTest.php [21:29:49] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [21:30:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.68, 9.76, 9.80 [21:30:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.39, 3.14, 2.90 [21:31:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.92, 8.84, 8.97 [21:32:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.12, 3.03, 2.89 [21:32:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.08, 9.24, 8.98 [21:33:25] miraheze/CreateWiki - Universal-Omega the build has errored. [21:37:13] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.45, 7.49, 8.32 [21:37:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:39:01] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.26, 1.94, 1.52 [21:40:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.89, 7.32, 8.15 [21:41:01] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.14, 1.66, 1.47 [21:41:06] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.47, 8.73, 8.63 [21:42:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:43:04] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/f79d47cab7ea...70510c78ee1d [21:43:05] [02miraheze/CreateWiki] 07Universal-Omega 0370510c7 - Update RemoteWikiTest.php [21:43:07] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [21:44:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.84, 8.10, 8.26 [21:46:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.39, 7.64, 8.49 [21:46:29] miraheze/CreateWiki - Universal-Omega the build has errored. [21:46:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.53, 7.28, 7.93 [21:46:51] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/70510c78ee1d...3ccc57695e2a [21:46:53] [02miraheze/CreateWiki] 07Universal-Omega 033ccc576 - Update RemoteWikiTest.php [21:46:54] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [21:47:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:49:43] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.16, 1.75, 1.39 [21:50:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.72, 9.04, 8.87 [21:51:04] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.03, 7.99, 8.40 [21:51:42] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.63, 1.70, 1.41 [21:51:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.88, 8.99, 8.39 [21:52:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.28, 9.88, 9.20 [21:53:20] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/3ccc57695e2a...0b5215348b45 [21:53:22] [02miraheze/CreateWiki] 07Universal-Omega 030b52153 - Update RemoteWikiTest.php [21:53:23] [02CreateWiki] 07Universal-Omega synchronize pull request 03#304: Add RemoteWiki phpunit tests - 13https://github.com/miraheze/CreateWiki/pull/304 [21:53:30] miraheze/CreateWiki - Universal-Omega the build passed. [21:53:45] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [21:53:55] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.48, 7.93, 8.07 [21:54:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.38, 9.31, 9.08 [21:57:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.33, 8.83, 8.45 [21:59:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:59:47] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.33, 7.81, 8.14 [22:02:26] miraheze/CreateWiki - Universal-Omega the build passed. [22:02:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.50, 8.86, 8.35 [22:04:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.33, 7.79, 8.50 [22:04:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:04:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.39, 8.29, 8.18 [22:10:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.89, 3.43, 2.72 [22:10:40] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.12, 8.54, 8.34 [22:12:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.46, 2.97, 2.64 [22:18:32] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.61, 8.11, 8.37 [22:19:42] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.97, 1.60, 1.31 [22:20:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.38, 8.54, 8.21 [22:21:42] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.22, 1.43, 1.28 [22:22:18] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.68, 8.69, 8.10 [22:22:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.10, 8.91, 8.38 [22:23:00] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [22:23:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:24:13] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.82, 8.00, 7.92 [22:24:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.04, 9.08, 8.52 [22:25:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.19, 4.03, 3.18 [22:26:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.89, 8.01, 8.20 [22:28:06] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.79, 8.77, 8.27 [22:28:19] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.02, 8.74, 8.49 [22:30:01] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.84, 9.26, 8.49 [22:30:16] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.27, 8.06, 8.28 [22:31:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.89, 3.99, 3.46 [22:31:57] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.41, 8.78, 8.42 [22:33:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:33:52] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.50, 8.10, 8.22 [22:34:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.58, 3.64, 3.40 [22:38:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:38:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.79, 3.46, 3.01 [22:39:38] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.30, 1.71, 1.12 [22:40:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.90, 3.81, 3.18 [22:41:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.80, 8.66, 8.37 [22:41:38] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.03, 1.38, 1.06 [22:42:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.57, 3.16 [22:43:04] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.40, 8.40, 7.91 [22:43:31] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.39, 9.34, 8.66 [22:44:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.98, 3.83, 3.75 [22:45:04] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.61, 7.58, 7.66 [22:45:26] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.16, 8.93, 8.60 [22:46:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.13, 3.60, 3.25 [22:48:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.41, 3.22 [22:48:28] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.38, 9.13, 8.28 [22:50:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.74, 3.28, 3.20 [22:50:23] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.39, 9.49, 8.53 [22:53:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:54:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.35, 8.36, 8.34 [22:54:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.15, 2.93, 3.32 [23:00:01] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:03] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:50] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.92, 9.11, 8.78 [23:02:50] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.40, 8.81, 8.71 [23:05:15] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.99, 3.83, 3.44 [23:06:50] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.41, 8.02, 8.43 [23:07:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.52, 3.44, 3.36 [23:09:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.37, 3.17, 3.27 [23:12:22] !log [@test101] finished deploy of {'l10nupdate': True} to all - SUCCESS in 740s [23:12:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:18:42] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1118s [23:18:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:20:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.17, 8.63, 8.03 [23:20:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:24:19] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.63, 8.15, 7.97 [23:26:33] PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query. [23:30:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:31:52] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.88, 2.06, 1.37 [23:33:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.14, 1.71, 1.33 [23:35:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.70, 1.37, 1.25 [23:39:51] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.94, 8.67, 8.18 [23:40:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.31, 8.18, 7.38 [23:42:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:42:39] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.46, 7.88, 7.37 [23:44:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.53, 3.53, 2.87 [23:44:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.02, 2.48 [23:46:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.03, 2.60, 2.39 [23:46:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.58, 3.13, 2.80 [23:47:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.17, 8.47, 8.31 [23:47:39] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:47:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.32, 9.12, 8.26 [23:51:04] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.85, 1.47, 1.22 [23:51:20] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.46, 8.93, 8.58 [23:51:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.05, 8.11, 8.06 [23:53:01] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.11, 1.39, 1.22 [23:54:38] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:55:10] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.33, 8.19, 8.37 [23:59:38] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki