[00:02:15] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.25, 5.51, 5.41 [00:02:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.69, 3.42 [00:02:51] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.57, 22.39, 20.70 [00:02:52] PROBLEM - mem121 Puppet on mem121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:02:53] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 2 minutes ago with 18 failures. Failed resources (up to 3 shown): File[/etc/firejail/mediawiki-converters.profile],File[/usr/local/bin/mediawiki-firejail-ghostscript],File[/srv/mediawiki/robots.php],File[/srv/mediawiki/sitemap.php] [00:04:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.83, 5.67, 5.48 [00:04:47] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.44, 3.36 [00:04:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.06, 22.06, 20.79 [00:08:42] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.21, 4.08, 3.62 [00:08:52] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.10, 22.62, 21.25 [00:10:39] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.79, 3.97, 3.64 [00:10:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.77, 22.40, 21.32 [00:11:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [00:11:48] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:12:12] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:12:22] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:12:23] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:12:32] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:12:41] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.11, 6.66, 6.27 [00:13:06] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.92, 6.99, 6.42 [00:13:08] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:12] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:13:37] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:42] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.520 second response time [00:14:13] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.331 second response time [00:14:15] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.51, 5.84, 5.62 [00:14:20] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.973 second response time [00:14:34] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.97, 3.70 [00:14:34] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.415 second response time [00:14:37] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.18, 6.25, 6.18 [00:15:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.74, 6.54, 6.32 [00:15:12] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.155 second response time [00:15:14] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 7.053 second response time [00:15:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:15:44] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.068 second response time [00:16:29] PROBLEM - bacula2 Bacula Databases db11 on bacula2 is WARNING: WARNING: Full, 83645 files, 16.78GB, 2021-12-26 00:14:00 (1.1 weeks ago) [00:16:33] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.298 second response time [00:16:51] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.45, 24.61, 22.51 [00:18:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.82, 3.75, 3.69 [00:19:54] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [00:20:43] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:44] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:48] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:21:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:21:07] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:21:16] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:05] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:43] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.764 second response time [00:22:45] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.250 second response time [00:22:50] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.472 second response time [00:23:05] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.623 second response time [00:23:12] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.811 second response time [00:23:59] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 7.888 second response time [00:24:07] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.814 second response time [00:25:07] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 25.15, 10.67, 6.04 [00:26:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.60, 3.60 [00:26:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:26:41] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.89, 5.87, 4.16 [00:26:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.52, 23.72, 23.64 [00:27:16] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:28:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.56, 3.58 [00:28:41] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.62, 5.00, 4.05 [00:28:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:29:16] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.301 second response time [00:30:37] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.15, 4.60, 3.59 [00:30:51] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.78, 23.60, 23.48 [00:30:52] RECOVERY - mem121 Puppet on mem121 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [00:30:52] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [00:30:52] PROBLEM - bacula2 Bacula Databases db12 on bacula2 is WARNING: WARNING: Full, 77875 files, 22.76GB, 2021-12-26 00:27:00 (1.1 weeks ago) [00:31:07] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 1.47, 4.51, 4.70 [00:32:32] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 2.14, 3.65, 3.36 [00:32:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.67, 20.85, 22.49 [00:34:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.48, 3.52 [00:36:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.55, 3.54 [00:36:51] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.35, 21.55, 22.30 [00:38:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.04, 20.60, 21.83 [00:39:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.57, 3.53 [00:41:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ShortDescription-d91fac8 [+0/-0/±1] 13https://git.io/JSZCJ [00:41:30] [02miraheze/mediawiki] 07dependabot[bot] 030132c1b - Bump extensions/ShortDescription from `543b282` to `d91fac8` [00:41:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ShortDescription-d91fac8 - 13https://git.io/vbL5b [00:41:33] [02mediawiki] 07dependabot[bot] opened pull request 03#4789: Bump extensions/ShortDescription from `543b282` to `d91fac8` - 13https://git.io/JSZCU [00:41:34] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Thanks-06f0c44 [+0/-0/±1] 13https://git.io/JSZCI [00:41:36] [02miraheze/mediawiki] 07dependabot[bot] 033f4fca7 - Bump extensions/Thanks from `525c1ba` to `06f0c44` [00:41:37] [02mediawiki] 07dependabot[bot] opened pull request 03#4790: Bump extensions/Thanks from `525c1ba` to `06f0c44` - 13https://git.io/JSZCL [00:41:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Thanks-06f0c44 - 13https://git.io/vbL5b [00:41:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ArticleCreationWorkflow-d3c2f81 [+0/-0/±1] 13https://git.io/JSZCq [00:41:42] [02miraheze/mediawiki] 07dependabot[bot] 03bdbbd96 - Bump extensions/ArticleCreationWorkflow from `f7965da` to `d3c2f81` [00:41:43] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ArticleCreationWorkflow-d3c2f81 - 13https://git.io/vbL5b [00:41:45] [02mediawiki] 07dependabot[bot] opened pull request 03#4791: Bump extensions/ArticleCreationWorkflow from `f7965da` to `d3c2f81` - 13https://git.io/JSZCm [00:41:46] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DiscussionTools-7209e5f [+0/-0/±1] 13https://git.io/JSZCY [00:41:48] [02miraheze/mediawiki] 07dependabot[bot] 03492bd88 - Bump extensions/DiscussionTools from `e9efabc` to `7209e5f` [00:41:49] [02mediawiki] 07dependabot[bot] opened pull request 03#4792: Bump extensions/DiscussionTools from `e9efabc` to `7209e5f` - 13https://git.io/JSZCO [00:41:51] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DiscussionTools-7209e5f - 13https://git.io/vbL5b [00:41:52] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Graph-676ba72 [+0/-0/±1] 13https://git.io/JSZCZ [00:41:54] [02miraheze/mediawiki] 07dependabot[bot] 036a88401 - Bump extensions/Graph from `f687b7d` to `676ba72` [00:41:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.65, 3.11, 3.36 [00:41:55] [02mediawiki] 07dependabot[bot] opened pull request 03#4793: Bump extensions/Graph from `f687b7d` to `676ba72` - 13https://git.io/JSZCn [00:41:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Graph-676ba72 - 13https://git.io/vbL5b [00:41:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MsCalendar-d30d8a9 [+0/-0/±1] 13https://git.io/JSZCC [00:42:00] [02miraheze/mediawiki] 07dependabot[bot] 03379b6d3 - Bump extensions/MsCalendar from `f0501ad` to `d30d8a9` [00:42:01] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MsCalendar-d30d8a9 - 13https://git.io/vbL5b [00:42:03] [02mediawiki] 07dependabot[bot] opened pull request 03#4794: Bump extensions/MsCalendar from `f0501ad` to `d30d8a9` - 13https://git.io/JSZCW [00:42:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AJAXPoll-2ea5085 [+0/-0/±1] 13https://git.io/JSZCl [00:42:06] [02miraheze/mediawiki] 07dependabot[bot] 0321354a0 - Bump extensions/AJAXPoll from `1c65c99` to `2ea5085` [00:42:07] [02mediawiki] 07dependabot[bot] opened pull request 03#4795: Bump extensions/AJAXPoll from `1c65c99` to `2ea5085` - 13https://git.io/JSZC8 [00:42:08] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 3.43, 5.24, 4.11 [00:42:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AJAXPoll-2ea5085 - 13https://git.io/vbL5b [00:42:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#4789: Bump extensions/ShortDescription from `543b282` to `d91fac8` - 13https://git.io/JSZCU [00:42:12] [02mediawiki] 07github-actions[bot] labeled pull request 03#4789: Bump extensions/ShortDescription from `543b282` to `d91fac8` - 13https://git.io/JSZCU [00:42:14] [02mediawiki] 07github-actions[bot] labeled pull request 03#4789: Bump extensions/ShortDescription from `543b282` to `d91fac8` - 13https://git.io/JSZCU [00:42:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-f33d7ce [+0/-0/±1] 13https://git.io/JSZC0 [00:42:16] [02miraheze/mediawiki] 07dependabot[bot] 03e4b2616 - Bump extensions/UniversalLanguageSelector from `1f6a7af` to `f33d7ce` [00:42:18] [02mediawiki] 07dependabot[bot] opened pull request 03#4796: Bump extensions/UniversalLanguageSelector from `1f6a7af` to `f33d7ce` - 13https://git.io/JSZCE [00:42:20] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-f33d7ce - 13https://git.io/vbL5b [00:42:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GeoData-9a7861c [+0/-0/±1] 13https://git.io/JSZCg [00:42:22] [02miraheze/mediawiki] 07dependabot[bot] 034c7927c - Bump extensions/GeoData from `ca332d7` to `9a7861c` [00:42:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GeoData-9a7861c - 13https://git.io/vbL5b [00:42:25] [02mediawiki] 07dependabot[bot] opened pull request 03#4797: Bump extensions/GeoData from `ca332d7` to `9a7861c` - 13https://git.io/JSZC2 [00:42:27] [02mediawiki] 07github-actions[bot] labeled pull request 03#4790: Bump extensions/Thanks from `525c1ba` to `06f0c44` - 13https://git.io/JSZCL [00:42:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalUsage-7071682 [+0/-0/±1] 13https://git.io/JSZCV [00:42:30] [02miraheze/mediawiki] 07dependabot[bot] 03dec9004 - Bump extensions/GlobalUsage from `1ed72d4` to `7071682` [00:42:32] [02mediawiki] 07dependabot[bot] opened pull request 03#4798: Bump extensions/GlobalUsage from `1ed72d4` to `7071682` - 13https://git.io/JSZCw [00:42:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalUsage-7071682 - 13https://git.io/vbL5b [00:42:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#4794: Bump extensions/MsCalendar from `f0501ad` to `d30d8a9` - 13https://git.io/JSZCW [00:42:36] [02mediawiki] 07github-actions[bot] labeled pull request 03#4791: Bump extensions/ArticleCreationWorkflow from `f7965da` to `d3c2f81` - 13https://git.io/JSZCm [00:42:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ArticlePlaceholder-d226109 [+0/-0/±1] 13https://git.io/JSZCK [00:42:39] [02miraheze/mediawiki] 07dependabot[bot] 0321d68eb - Bump extensions/ArticlePlaceholder from `bf21a5b` to `d226109` [00:42:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ArticlePlaceholder-d226109 - 13https://git.io/vbL5b [00:42:41] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 3.16, 5.36, 4.39 [00:42:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#4792: Bump extensions/DiscussionTools from `e9efabc` to `7209e5f` - 13https://git.io/JSZCO [00:42:43] [02mediawiki] 07dependabot[bot] opened pull request 03#4799: Bump extensions/ArticlePlaceholder from `bf21a5b` to `d226109` - 13https://git.io/JSZCi [00:42:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DynamicPageList-700a600 [+0/-0/±1] 13https://git.io/JSZCP [00:42:47] [02miraheze/mediawiki] 07dependabot[bot] 03835d0da - Bump extensions/DynamicPageList from `1ea15de` to `700a600` [00:42:48] [02mediawiki] 07dependabot[bot] opened pull request 03#4800: Bump extensions/DynamicPageList from `1ea15de` to `700a600` - 13https://git.io/JSZCX [00:42:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DynamicPageList-700a600 - 13https://git.io/vbL5b [00:42:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#4793: Bump extensions/Graph from `f687b7d` to `676ba72` - 13https://git.io/JSZCn [00:42:53] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Cosmos-95efb34 [+0/-0/±1] 13https://git.io/JSZCM [00:42:54] [02miraheze/mediawiki] 07dependabot[bot] 03e44fef0 - Bump skins/Cosmos from `2c46a3c` to `95efb34` [00:42:55] [02mediawiki] 07dependabot[bot] opened pull request 03#4801: Bump skins/Cosmos from `2c46a3c` to `95efb34` - 13https://git.io/JSZCD [00:42:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Cosmos-95efb34 - 13https://git.io/vbL5b [00:42:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#4795: Bump extensions/AJAXPoll from `1c65c99` to `2ea5085` - 13https://git.io/JSZC8 [00:43:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/wikihiero-f8c1e38 [+0/-0/±1] 13https://git.io/JSZCH [00:43:01] [02miraheze/mediawiki] 07dependabot[bot] 03537152d - Bump extensions/wikihiero from `3b35dd6` to `f8c1e38` [00:43:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/wikihiero-f8c1e38 - 13https://git.io/vbL5b [00:43:05] [02mediawiki] 07dependabot[bot] opened pull request 03#4802: Bump extensions/wikihiero from `3b35dd6` to `f8c1e38` - 13https://git.io/JSZCQ [00:43:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#4797: Bump extensions/GeoData from `ca332d7` to `9a7861c` - 13https://git.io/JSZC2 [00:43:08] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalWatchlist-d3cb716 [+0/-0/±1] 13https://git.io/JSZC7 [00:43:09] [02miraheze/mediawiki] 07dependabot[bot] 034211777 - Bump extensions/GlobalWatchlist from `6417a6d` to `d3cb716` [00:43:11] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalWatchlist-d3cb716 - 13https://git.io/vbL5b [00:43:12] [02mediawiki] 07dependabot[bot] opened pull request 03#4803: Bump extensions/GlobalWatchlist from `6417a6d` to `d3cb716` - 13https://git.io/JSZC5 [00:43:14] [02mediawiki] 07github-actions[bot] labeled pull request 03#4796: Bump extensions/UniversalLanguageSelector from `1f6a7af` to `f33d7ce` - 13https://git.io/JSZCE [00:43:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ProofreadPage-6878509 [+0/-0/±1] 13https://git.io/JSZCF [00:43:16] [02miraheze/mediawiki] 07dependabot[bot] 03bba4e78 - Bump extensions/ProofreadPage from `4c00d71` to `6878509` [00:43:18] [02mediawiki] 07dependabot[bot] opened pull request 03#4804: Bump extensions/ProofreadPage from `4c00d71` to `6878509` - 13https://git.io/JSZCb [00:43:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ProofreadPage-6878509 - 13https://git.io/vbL5b [00:43:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ManageWiki-a32b21d [+0/-0/±1] 13https://git.io/JSZCp [00:43:22] [02miraheze/mediawiki] 07dependabot[bot] 03af83611 - Bump extensions/ManageWiki from `cdda89f` to `a32b21d` [00:43:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ManageWiki-a32b21d - 13https://git.io/vbL5b [00:43:26] [02mediawiki] 07dependabot[bot] opened pull request 03#4805: Bump extensions/ManageWiki from `cdda89f` to `a32b21d` - 13https://git.io/JSZCh [00:43:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SkinPerPage-8512bf3 [+0/-0/±1] 13https://git.io/JSZCj [00:43:28] [02miraheze/mediawiki] 07dependabot[bot] 031a1228b - Bump extensions/SkinPerPage from `9fea664` to `8512bf3` [00:43:30] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SkinPerPage-8512bf3 - 13https://git.io/vbL5b [00:43:32] [02mediawiki] 07dependabot[bot] opened pull request 03#4806: Bump extensions/SkinPerPage from `9fea664` to `8512bf3` - 13https://git.io/JSZWe [00:43:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#4799: Bump extensions/ArticlePlaceholder from `bf21a5b` to `d226109` - 13https://git.io/JSZCi [00:43:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#4798: Bump extensions/GlobalUsage from `1ed72d4` to `7071682` - 13https://git.io/JSZCw [00:43:36] [02mediawiki] 07github-actions[bot] labeled pull request 03#4801: Bump skins/Cosmos from `2c46a3c` to `95efb34` - 13https://git.io/JSZCD [00:43:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AuthorProtect-0741a64 [+0/-0/±1] 13https://git.io/JSZWk [00:43:39] [02miraheze/mediawiki] 07dependabot[bot] 038cbdf31 - Bump extensions/AuthorProtect from `e1da8e9` to `0741a64` [00:43:41] [02mediawiki] 07dependabot[bot] opened pull request 03#4807: Bump extensions/AuthorProtect from `e1da8e9` to `0741a64` - 13https://git.io/JSZWI [00:43:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AuthorProtect-0741a64 - 13https://git.io/vbL5b [00:43:44] [02mediawiki] 07github-actions[bot] labeled pull request 03#4800: Bump extensions/DynamicPageList from `1ea15de` to `700a600` - 13https://git.io/JSZCX [00:43:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#4803: Bump extensions/GlobalWatchlist from `6417a6d` to `d3cb716` - 13https://git.io/JSZC5 [00:43:46] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageSchemas-7645ba0 [+0/-0/±1] 13https://git.io/JSZWL [00:43:48] [02miraheze/mediawiki] 07dependabot[bot] 0366918d7 - Bump extensions/PageSchemas from `ce663ab` to `7645ba0` [00:43:50] [02mediawiki] 07dependabot[bot] opened pull request 03#4808: Bump extensions/PageSchemas from `ce663ab` to `7645ba0` - 13https://git.io/JSZWt [00:43:51] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageSchemas-7645ba0 - 13https://git.io/vbL5b [00:43:52] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikibaseLexeme-3901a9b [+0/-0/±1] 13https://git.io/JSZWq [00:43:54] [02miraheze/mediawiki] 07dependabot[bot] 0357ffeb2 - Bump extensions/WikibaseLexeme from `e65c3d1` to `3901a9b` [00:43:56] [02mediawiki] 07dependabot[bot] opened pull request 03#4809: Bump extensions/WikibaseLexeme from `e65c3d1` to `3901a9b` - 13https://git.io/JSZWm [00:43:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikibaseLexeme-3901a9b - 13https://git.io/vbL5b [00:43:59] [02mediawiki] 07github-actions[bot] labeled pull request 03#4802: Bump extensions/wikihiero from `3b35dd6` to `f8c1e38` - 13https://git.io/JSZCQ [00:44:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#4804: Bump extensions/ProofreadPage from `4c00d71` to `6878509` - 13https://git.io/JSZCb [00:44:02] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Popups-dc1a397 [+0/-0/±1] 13https://git.io/JSZWs [00:44:03] [02miraheze/mediawiki] 07dependabot[bot] 0393c3987 - Bump extensions/Popups from `ac3468d` to `dc1a397` [00:44:05] [02mediawiki] 07dependabot[bot] opened pull request 03#4810: Bump extensions/Popups from `ac3468d` to `dc1a397` - 13https://git.io/JSZWG [00:44:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Popups-dc1a397 - 13https://git.io/vbL5b [00:44:07] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.39, 7.46, 5.18 [00:44:08] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/BetaFeatures-a8d6bdb [+0/-0/±1] 13https://git.io/JSZWn [00:44:09] [02miraheze/mediawiki] 07dependabot[bot] 031a20fae - Bump extensions/BetaFeatures from `8ce691e` to `a8d6bdb` [00:44:11] [02mediawiki] 07dependabot[bot] opened pull request 03#4811: Bump extensions/BetaFeatures from `8ce691e` to `a8d6bdb` - 13https://git.io/JSZWc [00:44:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/BetaFeatures-a8d6bdb - 13https://git.io/vbL5b [00:44:14] [02mediawiki] 07github-actions[bot] labeled pull request 03#4805: Bump extensions/ManageWiki from `cdda89f` to `a32b21d` - 13https://git.io/JSZCh [00:44:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#4806: Bump extensions/SkinPerPage from `9fea664` to `8512bf3` - 13https://git.io/JSZWe [00:44:17] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PollNY-1d8efb8 [+0/-0/±1] 13https://git.io/JSZW8 [00:44:18] [02miraheze/mediawiki] 07dependabot[bot] 03f278df1 - Bump extensions/PollNY from `c13c562` to `1d8efb8` [00:44:20] [02mediawiki] 07dependabot[bot] opened pull request 03#4812: Bump extensions/PollNY from `c13c562` to `1d8efb8` - 13https://git.io/JSZWB [00:44:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PollNY-1d8efb8 - 13https://git.io/vbL5b [00:44:23] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Echo-e2d6620 [+0/-0/±1] 13https://git.io/JSZWR [00:44:24] [02miraheze/mediawiki] 07dependabot[bot] 039825924 - Bump extensions/Echo from `3e00e04` to `e2d6620` [00:44:26] [02mediawiki] 07dependabot[bot] opened pull request 03#4813: Bump extensions/Echo from `3e00e04` to `e2d6620` - 13https://git.io/JSZW0 [00:44:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Echo-e2d6620 - 13https://git.io/vbL5b [00:44:29] [02mediawiki] 07github-actions[bot] labeled pull request 03#4807: Bump extensions/AuthorProtect from `e1da8e9` to `0741a64` - 13https://git.io/JSZWI [00:44:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TwoColConflict-783002e [+0/-0/±1] 13https://git.io/JSZWz [00:44:32] [02miraheze/mediawiki] 07dependabot[bot] 039d59a78 - Bump extensions/TwoColConflict from `184115a` to `783002e` [00:44:33] [02mediawiki] 07dependabot[bot] opened pull request 03#4814: Bump extensions/TwoColConflict from `184115a` to `783002e` - 13https://git.io/JSZWg [00:44:35] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TwoColConflict-783002e - 13https://git.io/vbL5b [00:44:36] [02mediawiki] 07github-actions[bot] labeled pull request 03#4808: Bump extensions/PageSchemas from `ce663ab` to `7645ba0` - 13https://git.io/JSZWt [00:44:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Metrolook-577b947 [+0/-0/±1] 13https://git.io/JSZWa [00:44:39] [02miraheze/mediawiki] 07dependabot[bot] 03415df53 - Bump skins/Metrolook from `5dc82cd` to `577b947` [00:44:41] [02mediawiki] 07dependabot[bot] opened pull request 03#4815: Bump skins/Metrolook from `5dc82cd` to `577b947` - 13https://git.io/JSZWV [00:44:41] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 1.77, 4.07, 4.03 [00:44:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Metrolook-577b947 - 13https://git.io/vbL5b [00:44:44] [02mediawiki] 07github-actions[bot] labeled pull request 03#4809: Bump extensions/WikibaseLexeme from `e65c3d1` to `3901a9b` - 13https://git.io/JSZWm [00:44:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/SocialProfile-f407fdb [+0/-0/±1] 13https://git.io/JSZWr [00:44:47] [02miraheze/mediawiki] 07dependabot[bot] 034569767 - Bump extensions/SocialProfile from `952f5d4` to `f407fdb` [00:44:48] [02mediawiki] 07dependabot[bot] opened pull request 03#4816: Bump extensions/SocialProfile from `952f5d4` to `f407fdb` - 13https://git.io/JSZWo [00:44:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/SocialProfile-f407fdb - 13https://git.io/vbL5b [00:44:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#4810: Bump extensions/Popups from `ac3468d` to `dc1a397` - 13https://git.io/JSZWG [00:44:53] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Comments-c390305 [+0/-0/±1] 13https://git.io/JSZWi [00:44:54] [02miraheze/mediawiki] 07dependabot[bot] 03dc3580b - Bump extensions/Comments from `f79ada2` to `c390305` [00:44:56] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Comments-c390305 - 13https://git.io/vbL5b [00:44:57] [02mediawiki] 07dependabot[bot] opened pull request 03#4817: Bump extensions/Comments from `f79ada2` to `c390305` - 13https://git.io/JSZWP [00:44:59] [02mediawiki] 07github-actions[bot] labeled pull request 03#4812: Bump extensions/PollNY from `c13c562` to `1d8efb8` - 13https://git.io/JSZWB [00:45:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CommonsMetadata-f06ab37 [+0/-0/±1] 13https://git.io/JSZW1 [00:45:02] [02miraheze/mediawiki] 07dependabot[bot] 03fa43378 - Bump extensions/CommonsMetadata from `94cf634` to `f06ab37` [00:45:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#4811: Bump extensions/BetaFeatures from `8ce691e` to `a8d6bdb` - 13https://git.io/JSZWc [00:45:05] [02mediawiki] 07dependabot[bot] opened pull request 03#4818: Bump extensions/CommonsMetadata from `94cf634` to `f06ab37` - 13https://git.io/JSZWM [00:45:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CommonsMetadata-f06ab37 - 13https://git.io/vbL5b [00:45:08] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Evelution-3d61d87 [+0/-0/±1] 13https://git.io/JSZWS [00:45:09] [02miraheze/mediawiki] 07dependabot[bot] 03dae97ac - Bump skins/Evelution from `da17274` to `3d61d87` [00:45:11] [02mediawiki] 07dependabot[bot] opened pull request 03#4819: Bump skins/Evelution from `da17274` to `3d61d87` - 13https://git.io/JSZWH [00:45:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Evelution-3d61d87 - 13https://git.io/vbL5b [00:45:14] [02mediawiki] 07github-actions[bot] labeled pull request 03#4813: Bump extensions/Echo from `3e00e04` to `e2d6620` - 13https://git.io/JSZW0 [00:45:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikibaseQualityConstraints-9752e3d [+0/-0/±1] 13https://git.io/JSZW7 [00:45:17] [02miraheze/mediawiki] 07dependabot[bot] 033ffcc32 - Bump extensions/WikibaseQualityConstraints from `4f9875a` to `9752e3d` [00:45:18] [02mediawiki] 07dependabot[bot] opened pull request 03#4820: Bump extensions/WikibaseQualityConstraints from `4f9875a` to `9752e3d` - 13https://git.io/JSZW5 [00:45:20] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikibaseQualityConstraints-9752e3d - 13https://git.io/vbL5b [00:45:21] [02mediawiki] 07github-actions[bot] labeled pull request 03#4814: Bump extensions/TwoColConflict from `184115a` to `783002e` - 13https://git.io/JSZWg [00:45:23] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/JsonConfig-20a48c3 [+0/-0/±1] 13https://git.io/JSZWF [00:45:24] [02miraheze/mediawiki] 07dependabot[bot] 0314ed4ba - Bump extensions/JsonConfig from `f23fb62` to `20a48c3` [00:45:26] [02mediawiki] 07dependabot[bot] opened pull request 03#4821: Bump extensions/JsonConfig from `f23fb62` to `20a48c3` - 13https://git.io/JSZWb [00:45:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/JsonConfig-20a48c3 - 13https://git.io/vbL5b [00:45:29] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/FlaggedRevs-796798a [+0/-0/±1] 13https://git.io/JSZWx [00:45:30] [02miraheze/mediawiki] 07dependabot[bot] 03ee9d327 - Bump extensions/FlaggedRevs from `91e026e` to `796798a` [00:45:32] [02mediawiki] 07dependabot[bot] opened pull request 03#4822: Bump extensions/FlaggedRevs from `91e026e` to `796798a` - 13https://git.io/JSZWp [00:45:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/FlaggedRevs-796798a - 13https://git.io/vbL5b [00:45:35] [02mediawiki] 07github-actions[bot] labeled pull request 03#4815: Bump skins/Metrolook from `5dc82cd` to `577b947` - 13https://git.io/JSZWV [00:45:36] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Wikibase-da93956 [+0/-0/±1] 13https://git.io/JSZWj [00:45:38] [02miraheze/mediawiki] 07dependabot[bot] 036ddb724 - Bump extensions/Wikibase from `ba8e322` to `da93956` [00:45:39] [02mediawiki] 07dependabot[bot] opened pull request 03#4823: Bump extensions/Wikibase from `ba8e322` to `da93956` - 13https://git.io/JSZle [00:45:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Wikibase-da93956 - 13https://git.io/vbL5b [00:45:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#4816: Bump extensions/SocialProfile from `952f5d4` to `f407fdb` - 13https://git.io/JSZWo [00:45:44] [02mediawiki] 07github-actions[bot] labeled pull request 03#4817: Bump extensions/Comments from `f79ada2` to `c390305` - 13https://git.io/JSZWP [00:45:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#4818: Bump extensions/CommonsMetadata from `94cf634` to `f06ab37` - 13https://git.io/JSZWM [00:45:46] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.04, 3.93, 3.62 [00:45:47] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MassMessage-bd6de25 [+0/-0/±1] 13https://git.io/JSZlT [00:45:48] [02miraheze/mediawiki] 07dependabot[bot] 0337fc044 - Bump extensions/MassMessage from `3a0d106` to `bd6de25` [00:45:50] [02mediawiki] 07dependabot[bot] opened pull request 03#4824: Bump extensions/MassMessage from `3a0d106` to `bd6de25` - 13https://git.io/JSZlk [00:45:51] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MassMessage-bd6de25 - 13https://git.io/vbL5b [00:45:53] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CentralNotice-d3d1795 [+0/-0/±1] 13https://git.io/JSZlI [00:45:54] [02miraheze/mediawiki] 07dependabot[bot] 0322ca461 - Bump extensions/CentralNotice from `d787a35` to `d3d1795` [00:45:56] [02mediawiki] 07dependabot[bot] opened pull request 03#4825: Bump extensions/CentralNotice from `d787a35` to `d3d1795` - 13https://git.io/JSZlL [00:45:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CentralNotice-d3d1795 - 13https://git.io/vbL5b [00:45:59] [02mediawiki] 07github-actions[bot] labeled pull request 03#4819: Bump skins/Evelution from `da17274` to `3d61d87` - 13https://git.io/JSZWH [00:46:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#4820: Bump extensions/WikibaseQualityConstraints from `4f9875a` to `9752e3d` - 13https://git.io/JSZW5 [00:46:02] [02mediawiki] 07github-actions[bot] labeled pull request 03#4819: Bump skins/Evelution from `da17274` to `3d61d87` - 13https://git.io/JSZWH [00:46:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Translate-941deb5 [+0/-0/±1] 13https://git.io/JSZlY [00:46:05] [02miraheze/mediawiki] 07dependabot[bot] 036b122cf - Bump extensions/Translate from `1c5f696` to `941deb5` [00:46:06] [02mediawiki] 07dependabot[bot] opened pull request 03#4826: Bump extensions/Translate from `1c5f696` to `941deb5` - 13https://git.io/JSZlO [00:46:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 2.25, 5.51, 4.74 [00:46:08] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Translate-941deb5 - 13https://git.io/vbL5b [00:46:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AddHTMLMetaAndTitle-d641e66 [+0/-0/±1] 13https://git.io/JSZls [00:46:11] [02miraheze/mediawiki] 07dependabot[bot] 03c86c259 - Bump extensions/AddHTMLMetaAndTitle from `631d970` to `d641e66` [00:46:12] [02mediawiki] 07github-actions[bot] labeled pull request 03#4821: Bump extensions/JsonConfig from `f23fb62` to `20a48c3` - 13https://git.io/JSZWb [00:46:14] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AddHTMLMetaAndTitle-d641e66 - 13https://git.io/vbL5b [00:46:15] [02mediawiki] 07dependabot[bot] opened pull request 03#4827: Bump extensions/AddHTMLMetaAndTitle from `631d970` to `d641e66` - 13https://git.io/JSZlG [00:46:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.19, 5.22, 5.90 [00:46:17] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/MinervaNeue-7beea49 [+0/-0/±1] 13https://git.io/JSZlZ [00:46:18] [02miraheze/mediawiki] 07dependabot[bot] 0379dcea9 - Bump skins/MinervaNeue from `75da362` to `7beea49` [00:46:20] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/MinervaNeue-7beea49 - 13https://git.io/vbL5b [00:46:21] [02mediawiki] 07dependabot[bot] opened pull request 03#4828: Bump skins/MinervaNeue from `75da362` to `7beea49` - 13https://git.io/JSZln [00:46:23] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DataDump-f47a800 [+0/-0/±1] 13https://git.io/JSZlW [00:46:24] [02miraheze/mediawiki] 07dependabot[bot] 0308c8335 - Bump extensions/DataDump from `bcb4503` to `f47a800` [00:46:26] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DataDump-f47a800 - 13https://git.io/vbL5b [00:46:27] [02mediawiki] 07dependabot[bot] opened pull request 03#4829: Bump extensions/DataDump from `bcb4503` to `f47a800` - 13https://git.io/JSZll [00:46:29] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PdfHandler-ba526c2 [+0/-0/±1] 13https://git.io/JSZl4 [00:46:30] [02miraheze/mediawiki] 07dependabot[bot] 03c3e5dd7 - Bump extensions/PdfHandler from `dcf936c` to `ba526c2` [00:46:32] [02mediawiki] 07dependabot[bot] opened pull request 03#4830: Bump extensions/PdfHandler from `dcf936c` to `ba526c2` - 13https://git.io/JSZlB [00:46:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PdfHandler-ba526c2 - 13https://git.io/vbL5b [00:46:35] [02mediawiki] 07github-actions[bot] labeled pull request 03#4824: Bump extensions/MassMessage from `3a0d106` to `bd6de25` - 13https://git.io/JSZlk [00:46:36] [02mediawiki] 07github-actions[bot] labeled pull request 03#4822: Bump extensions/FlaggedRevs from `91e026e` to `796798a` - 13https://git.io/JSZWp [00:46:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ApprovedRevs-609df70 [+0/-0/±1] 13https://git.io/JSZl0 [00:46:39] [02miraheze/mediawiki] 07dependabot[bot] 03aee78d2 - Bump extensions/ApprovedRevs from `ef1833a` to `609df70` [00:46:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ApprovedRevs-609df70 - 13https://git.io/vbL5b [00:46:42] [02mediawiki] 07dependabot[bot] opened pull request 03#4831: Bump extensions/ApprovedRevs from `ef1833a` to `609df70` - 13https://git.io/JSZlE [00:46:44] [02mediawiki] 07github-actions[bot] labeled pull request 03#4827: Bump extensions/AddHTMLMetaAndTitle from `631d970` to `d641e66` - 13https://git.io/JSZlG [00:46:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CentralAuth-d59e785 [+0/-0/±1] 13https://git.io/JSZlz [00:46:47] [02miraheze/mediawiki] 07dependabot[bot] 03a044c1f - Bump extensions/CentralAuth from `da5a133` to `d59e785` [00:46:48] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CentralAuth-d59e785 - 13https://git.io/vbL5b [00:46:50] [02mediawiki] 07github-actions[bot] labeled pull request 03#4825: Bump extensions/CentralNotice from `d787a35` to `d3d1795` - 13https://git.io/JSZlL [00:46:51] [02mediawiki] 07dependabot[bot] opened pull request 03#4832: Bump extensions/CentralAuth from `da5a133` to `d59e785` - 13https://git.io/JSZla [00:46:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.39, 18.53, 20.35 [00:46:53] [02mediawiki] 07github-actions[bot] labeled pull request 03#4823: Bump extensions/Wikibase from `ba8e322` to `da93956` - 13https://git.io/JSZle [00:46:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/JSBreadCrumbs-0c84de9 [+0/-0/±1] 13https://git.io/JSZlw [00:46:56] [02miraheze/mediawiki] 07dependabot[bot] 030b7eee4 - Bump extensions/JSBreadCrumbs from `0c9d734` to `0c84de9` [00:46:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/JSBreadCrumbs-0c84de9 - 13https://git.io/vbL5b [00:46:59] [02mediawiki] 07dependabot[bot] opened pull request 03#4833: Bump extensions/JSBreadCrumbs from `0c9d734` to `0c84de9` - 13https://git.io/JSZlr [00:47:00] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/chameleon-4428398 [+0/-0/±1] 13https://git.io/JSZlK [00:47:02] [02miraheze/mediawiki] 07dependabot[bot] 03c03fe46 - Bump skins/chameleon from `a923c46` to `4428398` [00:47:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/chameleon-4428398 - 13https://git.io/vbL5b [00:47:05] [02mediawiki] 07dependabot[bot] opened pull request 03#4834: Bump skins/chameleon from `a923c46` to `4428398` - 13https://git.io/JSZl6 [00:47:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#4828: Bump skins/MinervaNeue from `75da362` to `7beea49` - 13https://git.io/JSZln [00:47:08] [02mediawiki] 07github-actions[bot] labeled pull request 03#4829: Bump extensions/DataDump from `bcb4503` to `f47a800` - 13https://git.io/JSZll [00:47:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Disambiguator-53e8268 [+0/-0/±1] 13https://git.io/JSZlP [00:47:11] [02miraheze/mediawiki] 07dependabot[bot] 037d6cebe - Bump extensions/Disambiguator from `9b870e9` to `53e8268` [00:47:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Disambiguator-53e8268 - 13https://git.io/vbL5b [00:47:14] [02mediawiki] 07github-actions[bot] labeled pull request 03#4826: Bump extensions/Translate from `1c5f696` to `941deb5` - 13https://git.io/JSZlO [00:47:15] [02mediawiki] 07dependabot[bot] opened pull request 03#4835: Bump extensions/Disambiguator from `9b870e9` to `53e8268` - 13https://git.io/JSZlX [00:47:17] [02mediawiki] 07github-actions[bot] labeled pull request 03#4826: Bump extensions/Translate from `1c5f696` to `941deb5` - 13https://git.io/JSZlO [00:47:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageTriage-5b81b50 [+0/-0/±1] 13https://git.io/JSZlM [00:47:20] [02miraheze/mediawiki] 07dependabot[bot] 038aa7ccf - Bump extensions/PageTriage from `459e8fe` to `5b81b50` [00:47:21] [02mediawiki] 07dependabot[bot] opened pull request 03#4836: Bump extensions/PageTriage from `459e8fe` to `5b81b50` - 13https://git.io/JSZlD [00:47:23] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageTriage-5b81b50 - 13https://git.io/vbL5b [00:47:24] [02mediawiki] 07github-actions[bot] labeled pull request 03#4830: Bump extensions/PdfHandler from `dcf936c` to `ba526c2` - 13https://git.io/JSZlB [00:47:26] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TimedMediaHandler-ad5c67f [+0/-0/±1] 13https://git.io/JSZlH [00:47:27] [02miraheze/mediawiki] 07dependabot[bot] 039115664 - Bump extensions/TimedMediaHandler from `b2577ee` to `ad5c67f` [00:47:29] [02mediawiki] 07dependabot[bot] opened pull request 03#4837: Bump extensions/TimedMediaHandler from `b2577ee` to `ad5c67f` - 13https://git.io/JSZlQ [00:47:30] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TimedMediaHandler-ad5c67f - 13https://git.io/vbL5b [00:47:32] [02mediawiki] 07github-actions[bot] labeled pull request 03#4831: Bump extensions/ApprovedRevs from `ef1833a` to `609df70` - 13https://git.io/JSZlE [00:47:32] PROBLEM - bacula2 Bacula Databases db13 on bacula2 is WARNING: WARNING: Full, 77836 files, 29.60GB, 2021-12-26 00:44:00 (1.1 weeks ago) [00:47:33] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Nimbus-7b47d15 [+0/-0/±1] 13https://git.io/JSZl7 [00:47:35] [02miraheze/mediawiki] 07dependabot[bot] 03630ec8e - Bump skins/Nimbus from `10aba31` to `7b47d15` [00:47:36] [02mediawiki] 07dependabot[bot] opened pull request 03#4838: Bump skins/Nimbus from `10aba31` to `7b47d15` - 13https://git.io/JSZl5 [00:47:38] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Nimbus-7b47d15 - 13https://git.io/vbL5b [00:47:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PagedTiffHandler-5c2fe18 [+0/-0/±1] 13https://git.io/JSZlN [00:47:41] [02miraheze/mediawiki] 07dependabot[bot] 03d6a9120 - Bump extensions/PagedTiffHandler from `1ca78be` to `5c2fe18` [00:47:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PagedTiffHandler-5c2fe18 - 13https://git.io/vbL5b [00:47:44] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.97, 3.68 [00:47:44] [02mediawiki] 07dependabot[bot] opened pull request 03#4839: Bump extensions/PagedTiffHandler from `1ca78be` to `5c2fe18` - 13https://git.io/JSZlA [00:47:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#4832: Bump extensions/CentralAuth from `da5a133` to `d59e785` - 13https://git.io/JSZla [00:47:47] [02mediawiki] 07github-actions[bot] labeled pull request 03#4833: Bump extensions/JSBreadCrumbs from `0c9d734` to `0c84de9` - 13https://git.io/JSZlr [00:47:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#4835: Bump extensions/Disambiguator from `9b870e9` to `53e8268` - 13https://git.io/JSZlX [00:47:50] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/UrlShortener-820b555 [+0/-0/±1] 13https://git.io/JSZlp [00:47:51] [02miraheze/mediawiki] 07dependabot[bot] 03845aa6c - Bump extensions/UrlShortener from `cb19751` to `820b555` [00:47:53] [02mediawiki] 07dependabot[bot] opened pull request 03#4840: Bump extensions/UrlShortener from `cb19751` to `820b555` - 13https://git.io/JSZlh [00:47:54] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/UrlShortener-820b555 - 13https://git.io/vbL5b [00:47:56] [02mediawiki] 07github-actions[bot] labeled pull request 03#4834: Bump skins/chameleon from `a923c46` to `4428398` - 13https://git.io/JSZl6 [00:47:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#4834: Bump skins/chameleon from `a923c46` to `4428398` - 13https://git.io/JSZl6 [00:47:59] [02mediawiki] 07github-actions[bot] labeled pull request 03#4834: Bump skins/chameleon from `a923c46` to `4428398` - 13https://git.io/JSZl6 [00:48:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#4837: Bump extensions/TimedMediaHandler from `b2577ee` to `ad5c67f` - 13https://git.io/JSZlQ [00:48:02] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MobileFrontend-8f64d85 [+0/-0/±1] 13https://git.io/JSZ8v [00:48:03] [02miraheze/mediawiki] 07dependabot[bot] 03a552b07 - Bump extensions/MobileFrontend from `6c4bee5` to `8f64d85` [00:48:05] [02mediawiki] 07dependabot[bot] opened pull request 03#4841: Bump extensions/MobileFrontend from `6c4bee5` to `8f64d85` - 13https://git.io/JSZ8f [00:48:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MobileFrontend-8f64d85 - 13https://git.io/vbL5b [00:48:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 2.24, 4.52, 4.47 [00:48:08] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PortableInfobox-b0a072f [+0/-0/±1] 13https://git.io/JSZ8T [00:48:09] [02miraheze/mediawiki] 07dependabot[bot] 037f7ef1b - Bump extensions/PortableInfobox from `c934b05` to `b0a072f` [00:48:11] [02mediawiki] 07github-actions[bot] labeled pull request 03#4836: Bump extensions/PageTriage from `459e8fe` to `5b81b50` - 13https://git.io/JSZlD [00:48:12] [02mediawiki] 07dependabot[bot] opened pull request 03#4842: Bump extensions/PortableInfobox from `c934b05` to `b0a072f` - 13https://git.io/JSZ8k [00:48:14] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PortableInfobox-b0a072f - 13https://git.io/vbL5b [00:48:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageForms-7c19d92 [+0/-0/±1] 13https://git.io/JSZ8L [00:48:17] [02miraheze/mediawiki] 07dependabot[bot] 0364d9c6f - Bump extensions/PageForms from `1742de7` to `7c19d92` [00:48:18] [02mediawiki] 07dependabot[bot] opened pull request 03#4843: Bump extensions/PageForms from `1742de7` to `7c19d92` - 13https://git.io/JSZ8q [00:48:20] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageForms-7c19d92 - 13https://git.io/vbL5b [00:48:21] [02mediawiki] 07github-actions[bot] labeled pull request 03#4838: Bump skins/Nimbus from `10aba31` to `7b47d15` - 13https://git.io/JSZl5 [00:48:23] [02mediawiki] 07dependabot[bot] opened pull request 03#4844: Bump extensions/Score from `bd4416e` to `932fe9e` - 13https://git.io/JSZ8Y [00:48:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Score-932fe9e [+0/-0/±1] 13https://git.io/JSZ8O [00:48:26] [02miraheze/mediawiki] 07dependabot[bot] 03c5ce1d1 - Bump extensions/Score from `bd4416e` to `932fe9e` [00:48:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Score-932fe9e - 13https://git.io/vbL5b [00:48:29] [02mediawiki] 07github-actions[bot] labeled pull request 03#4839: Bump extensions/PagedTiffHandler from `1ca78be` to `5c2fe18` - 13https://git.io/JSZlA [00:48:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TemplateSandbox-d7ac520 [+0/-0/±1] 13https://git.io/JSZ8G [00:48:32] [02miraheze/mediawiki] 07dependabot[bot] 03ba1a485 - Bump extensions/TemplateSandbox from `7ebf3bd` to `d7ac520` [00:48:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TemplateSandbox-d7ac520 - 13https://git.io/vbL5b [00:48:35] [02mediawiki] 07dependabot[bot] opened pull request 03#4845: Bump extensions/TemplateSandbox from `7ebf3bd` to `d7ac520` - 13https://git.io/JSZ8Z [00:48:36] [02mediawiki] 07github-actions[bot] labeled pull request 03#4840: Bump extensions/UrlShortener from `cb19751` to `820b555` - 13https://git.io/JSZlh [00:48:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikimediaIncubator-12b9543 [+0/-0/±1] 13https://git.io/JSZ8C [00:48:39] [02miraheze/mediawiki] 07dependabot[bot] 0370391eb - Bump extensions/WikimediaIncubator from `e63ab34` to `12b9543` [00:48:41] [02mediawiki] 07github-actions[bot] labeled pull request 03#4842: Bump extensions/PortableInfobox from `c934b05` to `b0a072f` - 13https://git.io/JSZ8k [00:48:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikimediaIncubator-12b9543 - 13https://git.io/vbL5b [00:48:44] [02mediawiki] 07dependabot[bot] opened pull request 03#4846: Bump extensions/WikimediaIncubator from `e63ab34` to `12b9543` - 13https://git.io/JSZ8W [00:48:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/UserMerge-f8306f2 [+0/-0/±1] 13https://git.io/JSZ88 [00:48:47] [02miraheze/mediawiki] 07dependabot[bot] 0377fe715 - Bump extensions/UserMerge from `57f440b` to `f8306f2` [00:48:48] [02mediawiki] 07dependabot[bot] opened pull request 03#4847: Bump extensions/UserMerge from `57f440b` to `f8306f2` - 13https://git.io/JSZ84 [00:48:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/UserMerge-f8306f2 - 13https://git.io/vbL5b [00:48:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#4841: Bump extensions/MobileFrontend from `6c4bee5` to `8f64d85` - 13https://git.io/JSZ8f [00:48:53] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Babel-9e55da4 [+0/-0/±1] 13https://git.io/JSZ80 [00:48:54] [02miraheze/mediawiki] 07dependabot[bot] 03b60811c - Bump extensions/Babel from `1334838` to `9e55da4` [00:48:56] [02mediawiki] 07dependabot[bot] opened pull request 03#4848: Bump extensions/Babel from `1334838` to `9e55da4` - 13https://git.io/JSZ8E [00:48:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Babel-9e55da4 - 13https://git.io/vbL5b [00:48:59] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AbuseFilter-4bf6a6a [+0/-0/±1] 13https://git.io/JSZ8g [00:49:00] [02miraheze/mediawiki] 07dependabot[bot] 03529e4c7 - Bump extensions/AbuseFilter from `b36e6a2` to `4bf6a6a` [00:49:02] [02mediawiki] 07dependabot[bot] opened pull request 03#4849: Bump extensions/AbuseFilter from `b36e6a2` to `4bf6a6a` - 13https://git.io/JSZ82 [00:49:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AbuseFilter-4bf6a6a - 13https://git.io/vbL5b [00:49:05] [02mediawiki] 07github-actions[bot] labeled pull request 03#4843: Bump extensions/PageForms from `1742de7` to `7c19d92` - 13https://git.io/JSZ8q [00:49:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#4843: Bump extensions/PageForms from `1742de7` to `7c19d92` - 13https://git.io/JSZ8q [00:49:08] [02mediawiki] 07github-actions[bot] labeled pull request 03#4844: Bump extensions/Score from `bd4416e` to `932fe9e` - 13https://git.io/JSZ8Y [00:49:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageAssessments-851096d [+0/-0/±1] 13https://git.io/JSZ8a [00:49:11] [02miraheze/mediawiki] 07dependabot[bot] 032b06e35 - Bump extensions/PageAssessments from `14d018f` to `851096d` [00:49:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageAssessments-851096d - 13https://git.io/vbL5b [00:49:14] [02mediawiki] 07dependabot[bot] opened pull request 03#4850: Bump extensions/PageAssessments from `14d018f` to `851096d` - 13https://git.io/JSZ8V [00:49:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#4845: Bump extensions/TemplateSandbox from `7ebf3bd` to `d7ac520` - 13https://git.io/JSZ8Z [00:49:17] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ContactPage-d5a4827 [+0/-0/±1] 13https://git.io/JSZ8K [00:49:18] [02miraheze/mediawiki] 07dependabot[bot] 03a55f0b7 - Bump extensions/ContactPage from `495144d` to `d5a4827` [00:49:20] [02mediawiki] 07dependabot[bot] opened pull request 03#4851: Bump extensions/ContactPage from `495144d` to `d5a4827` - 13https://git.io/JSZ86 [00:49:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ContactPage-d5a4827 - 13https://git.io/vbL5b [00:49:23] [02mediawiki] 07github-actions[bot] labeled pull request 03#4847: Bump extensions/UserMerge from `57f440b` to `f8306f2` - 13https://git.io/JSZ84 [00:49:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/LoginNotify-f5e4e43 [+0/-0/±1] 13https://git.io/JSZ8P [00:49:26] [02miraheze/mediawiki] 07dependabot[bot] 03a204bf3 - Bump extensions/LoginNotify from `3d35729` to `f5e4e43` [00:49:27] [02mediawiki] 07dependabot[bot] opened pull request 03#4852: Bump extensions/LoginNotify from `3d35729` to `f5e4e43` - 13https://git.io/JSZ8X [00:49:29] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/LoginNotify-f5e4e43 - 13https://git.io/vbL5b [00:49:30] PROBLEM - bacula2 Bacula Private Git on bacula2 is WARNING: WARNING: Full, 6817 files, 57.43MB, 2021-12-26 00:46:00 (1.1 weeks ago) [00:49:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#4846: Bump extensions/WikimediaIncubator from `e63ab34` to `12b9543` - 13https://git.io/JSZ8W [00:49:32] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikiDiscover-a70309d [+0/-0/±1] 13https://git.io/JSZ81 [00:49:33] [02miraheze/mediawiki] 07dependabot[bot] 03de05f04 - Bump extensions/WikiDiscover from `83eeb61` to `a70309d` [00:49:35] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikiDiscover-a70309d - 13https://git.io/vbL5b [00:49:36] [02mediawiki] 07dependabot[bot] opened pull request 03#4853: Bump extensions/WikiDiscover from `83eeb61` to `a70309d` - 13https://git.io/JSZ8M [00:49:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AntiSpoof-56b4b17 [+0/-0/±1] 13https://git.io/JSZ8H [00:49:39] [02miraheze/mediawiki] 07dependabot[bot] 03ed83d5b - Bump extensions/AntiSpoof from `a309821` to `56b4b17` [00:49:41] [02mediawiki] 07dependabot[bot] opened pull request 03#4854: Bump extensions/AntiSpoof from `a309821` to `56b4b17` - 13https://git.io/JSZ8Q [00:49:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AntiSpoof-56b4b17 - 13https://git.io/vbL5b [00:49:44] [02mediawiki] 07github-actions[bot] labeled pull request 03#4848: Bump extensions/Babel from `1334838` to `9e55da4` - 13https://git.io/JSZ8E [00:49:45] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Flow-fd89ee0 [+0/-0/±1] 13https://git.io/JSZ8d [00:49:47] [02miraheze/mediawiki] 07dependabot[bot] 03902e566 - Bump extensions/Flow from `f7ce11c` to `fd89ee0` [00:49:48] [02mediawiki] 07dependabot[bot] opened pull request 03#4855: Bump extensions/Flow from `f7ce11c` to `fd89ee0` - 13https://git.io/JSZ8F [00:49:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Flow-fd89ee0 - 13https://git.io/vbL5b [00:49:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#4850: Bump extensions/PageAssessments from `14d018f` to `851096d` - 13https://git.io/JSZ8V [00:49:53] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Tweeki-b88f5d7 [+0/-0/±1] 13https://git.io/JSZ8A [00:49:54] [02miraheze/mediawiki] 07dependabot[bot] 0324ef921 - Bump skins/Tweeki from `e5f76bc` to `b88f5d7` [00:49:56] [02mediawiki] 07dependabot[bot] opened pull request 03#4856: Bump skins/Tweeki from `e5f76bc` to `b88f5d7` - 13https://git.io/JSZ8x [00:49:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Tweeki-b88f5d7 - 13https://git.io/vbL5b [00:49:59] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/HAWelcome-dbc3826 [+0/-0/±1] 13https://git.io/JSZ8p [00:50:00] [02miraheze/mediawiki] 07dependabot[bot] 03a7a5f7d - Bump extensions/HAWelcome from `901f5c0` to `dbc3826` [00:50:02] [02mediawiki] 07dependabot[bot] opened pull request 03#4857: Bump extensions/HAWelcome from `901f5c0` to `dbc3826` - 13https://git.io/JSZ8h [00:50:03] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/HAWelcome-dbc3826 - 13https://git.io/vbL5b [00:50:05] [02mediawiki] 07github-actions[bot] labeled pull request 03#4851: Bump extensions/ContactPage from `495144d` to `d5a4827` - 13https://git.io/JSZ86 [00:50:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#4853: Bump extensions/WikiDiscover from `83eeb61` to `a70309d` - 13https://git.io/JSZ8M [00:50:08] [02mediawiki] 07github-actions[bot] labeled pull request 03#4854: Bump extensions/AntiSpoof from `a309821` to `56b4b17` - 13https://git.io/JSZ8Q [00:50:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Kartographer-819f44d [+0/-0/±1] 13https://git.io/JSZ4f [00:50:11] [02miraheze/mediawiki] 07dependabot[bot] 03b1b36a7 - Bump extensions/Kartographer from `a2d15e6` to `819f44d` [00:50:12] [02mediawiki] 07dependabot[bot] opened pull request 03#4858: Bump extensions/Kartographer from `a2d15e6` to `819f44d` - 13https://git.io/JSZ4J [00:50:14] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Kartographer-819f44d - 13https://git.io/vbL5b [00:50:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#4849: Bump extensions/AbuseFilter from `b36e6a2` to `4bf6a6a` - 13https://git.io/JSZ82 [00:50:17] [02mediawiki] 07github-actions[bot] labeled pull request 03#4852: Bump extensions/LoginNotify from `3d35729` to `f5e4e43` - 13https://git.io/JSZ8X [00:50:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CheckUser-b2255f3 [+0/-0/±1] 13https://git.io/JSZ4k [00:50:20] [02miraheze/mediawiki] 07dependabot[bot] 03bcbcd07 - Bump extensions/CheckUser from `db83b3f` to `b2255f3` [00:50:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CheckUser-b2255f3 - 13https://git.io/vbL5b [00:50:23] [02mediawiki] 07dependabot[bot] opened pull request 03#4859: Bump extensions/CheckUser from `db83b3f` to `b2255f3` - 13https://git.io/JSZ4I [00:50:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Femiwiki-21a54f2 [+0/-0/±1] 13https://git.io/JSZ4L [00:50:26] [02miraheze/mediawiki] 07dependabot[bot] 03780e841 - Bump skins/Femiwiki from `811ebbc` to `21a54f2` [00:50:27] [02mediawiki] 07dependabot[bot] opened pull request 03#4860: Bump skins/Femiwiki from `811ebbc` to `21a54f2` - 13https://git.io/JSZ4t [00:50:29] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Femiwiki-21a54f2 - 13https://git.io/vbL5b [00:50:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Cargo-07f465d [+0/-0/±1] 13https://git.io/JSZ4O [00:50:32] [02miraheze/mediawiki] 07dependabot[bot] 0330fc951 - Bump extensions/Cargo from `f7ba256` to `07f465d` [00:50:33] [02mediawiki] 07dependabot[bot] opened pull request 03#4861: Bump extensions/Cargo from `f7ba256` to `07f465d` - 13https://git.io/JSZ43 [00:50:35] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Cargo-07f465d - 13https://git.io/vbL5b [00:50:36] [02mediawiki] 07dependabot[bot] opened pull request 03#4862: Bump extensions/Newsletter from `53c9f52` to `3de7955` - 13https://git.io/JSZ4G [00:50:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Newsletter-3de7955 [+0/-0/±1] 13https://git.io/JSZ4Z [00:50:39] [02miraheze/mediawiki] 07dependabot[bot] 031cfa40f - Bump extensions/Newsletter from `53c9f52` to `3de7955` [00:50:41] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Newsletter-3de7955 - 13https://git.io/vbL5b [00:50:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#4855: Bump extensions/Flow from `f7ce11c` to `fd89ee0` - 13https://git.io/JSZ8F [00:50:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MassEditRegex-27b4608 [+0/-0/±1] 13https://git.io/JSZ4n [00:50:45] [02miraheze/mediawiki] 07dependabot[bot] 035a4cb14 - Bump extensions/MassEditRegex from `cf4a302` to `27b4608` [00:50:47] [02mediawiki] 07dependabot[bot] opened pull request 03#4863: Bump extensions/MassEditRegex from `cf4a302` to `27b4608` - 13https://git.io/JSZ4c [00:50:48] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MassEditRegex-27b4608 - 13https://git.io/vbL5b [00:50:50] [02mediawiki] 07github-actions[bot] labeled pull request 03#4856: Bump skins/Tweeki from `e5f76bc` to `b88f5d7` - 13https://git.io/JSZ8x [00:50:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.49, 19.69, 20.46 [00:50:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#4857: Bump extensions/HAWelcome from `901f5c0` to `dbc3826` - 13https://git.io/JSZ8h [00:50:53] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-8a068e8 [+0/-0/±1] 13https://git.io/JSZ4C [00:50:54] [02miraheze/mediawiki] 07dependabot[bot] 033538fac - Bump extensions/MirahezeMagic from `b1e2aed` to `8a068e8` [00:50:56] [02mediawiki] 07dependabot[bot] opened pull request 03#4864: Bump extensions/MirahezeMagic from `b1e2aed` to `8a068e8` - 13https://git.io/JSZ4W [00:50:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-8a068e8 - 13https://git.io/vbL5b [00:50:59] [02mediawiki] 07github-actions[bot] labeled pull request 03#4859: Bump extensions/CheckUser from `db83b3f` to `b2255f3` - 13https://git.io/JSZ4I [00:51:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#4858: Bump extensions/Kartographer from `a2d15e6` to `819f44d` - 13https://git.io/JSZ4J [00:51:02] [02mediawiki] 07github-actions[bot] labeled pull request 03#4860: Bump skins/Femiwiki from `811ebbc` to `21a54f2` - 13https://git.io/JSZ4t [00:51:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#4861: Bump extensions/Cargo from `f7ba256` to `07f465d` - 13https://git.io/JSZ43 [00:51:05] [02mediawiki] 07github-actions[bot] labeled pull request 03#4862: Bump extensions/Newsletter from `53c9f52` to `3de7955` - 13https://git.io/JSZ4G [00:51:06] [02mediawiki] 07github-actions[bot] labeled pull request 03#4863: Bump extensions/MassEditRegex from `cf4a302` to `27b4608` - 13https://git.io/JSZ4c [00:51:08] [02mediawiki] 07github-actions[bot] labeled pull request 03#4864: Bump extensions/MirahezeMagic from `b1e2aed` to `8a068e8` - 13https://git.io/JSZ4W [00:51:09] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb [00:51:38] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.95, 3.72 [00:52:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.88, 19.08, 20.14 [00:53:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:57:07] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 2.84, 5.98, 4.65 [00:57:13] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb [00:57:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.29, 3.70, 3.76 [00:59:07] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 2.02, 4.66, 4.33 [00:59:13] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:02:15] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.55, 4.50, 5.02 [01:03:07] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 2.31, 5.81, 5.24 [01:05:07] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 1.80, 4.50, 4.83 [01:05:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.47, 3.59 [01:06:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.10, 5.44, 5.34 [01:06:25] [02puppet] 07Universal-Omega opened pull request 03#2247: mwscript: fix parsing of arguments - 13https://git.io/JSZgj [01:06:58] paladox, if around mind merging ^? [01:07:50] [02puppet] 07paladox closed pull request 03#2247: mwscript: fix parsing of arguments - 13https://git.io/JSZgj [01:07:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSZ2z [01:07:53] [02miraheze/puppet] 07Universal-Omega 0330f604b - mwscript: fix parsing of arguments (#2247) [01:07:58] Thanks! [01:08:16] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.91, 6.36, 5.69 [01:09:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.47, 3.67, 3.65 [01:12:54] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:13:05] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [01:13:10] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [01:13:33] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.16, 6.76, 6.33 [01:13:37] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:13:37] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:13:58] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:14:35] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:14:56] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.718 second response time [01:15:29] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.79, 6.66, 6.33 [01:15:31] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.006 second response time [01:15:37] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.314 second response time [01:15:55] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.337 second response time [01:16:32] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 3.038 second response time [01:18:27] !log [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php purge (START) [01:19:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.88, 2.90, 3.33 [01:19:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:20:05] CosmicAlpha: oh, you have to do them one at a time, and log when each finishes? [01:20:22] dmehus: the script will log [01:20:30] CosmicAlpha, ah, cool :) [01:20:33] but yes one at a time. [01:20:52] I just had to fix the script first as it was broken that is why it took so long [01:20:56] is databases.json updated automatically by CreateWiki, and is it updated after each wiki creation or on a cron? [01:21:05] CosmicAlpha, ah, ok [01:21:21] yes on creation, by CreateWiki [01:21:22] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:21:25] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:21:29] CosmicAlpha: ah, cool :) [01:21:38] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:21:38] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:21:41] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:21:42] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:21:48] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:21:53] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:21:54] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:01] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:09] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:15] dmehus I've sent you a message on a private chat, although I have said it on the #cvt channel, so I won't have to repeat myself any further. [01:22:19] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:25] DarkMatterMan4500, will look yeah [01:22:41] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:45] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:48] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:51] Fair enough. At least for the both of us. [01:23:38] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.008 second response time [01:23:41] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.10, 1.99, 1.30 [01:23:44] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.331 second response time [01:23:49] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.337 second response time [01:23:55] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:23:58] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.066 second response time [01:24:01] @DarkMatterMan4500, replied to DM - checking -cvt [01:24:14] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.004 second response time [01:24:19] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.055 second response time [01:24:20] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:24:29] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:24:35] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:24:44] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 1.724 second response time [01:24:45] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.148 second response time [01:24:46] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 3.596 second response time [01:24:49] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:25:00] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.43, 22.42, 21.06 [01:25:29] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.305 second response time [01:25:39] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:25:40] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.70, 1.48, 1.20 [01:25:44] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.216 second response time [01:25:49] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 9.490 second response time [01:25:53] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.142 second response time [01:25:56] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.167 second response time [01:26:17] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.348 second response time [01:26:27] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.320 second response time [01:26:29] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:26:34] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.043 second response time [01:26:46] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 2.280 second response time [01:27:39] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.893 second response time [01:27:47] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.613 second response time [01:28:01] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.598 second response time [01:29:32] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.71, 6.74, 6.10 [01:30:29] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.63, 6.75, 5.54 [01:31:29] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:32:14] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:32:29] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.64, 6.28, 5.51 [01:32:48] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.455 second response time [01:33:21] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.14, 7.18, 6.40 [01:33:31] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.056 second response time [01:34:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.30, 4.60, 3.10 [01:34:17] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:34:21] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.169 second response time [01:34:37] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:34:51] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.76, 21.92, 21.28 [01:35:00] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:35:15] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.78, 7.21, 6.51 [01:36:07] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 19.22, 8.28, 4.51 [01:36:52] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.43, 22.73, 21.69 [01:37:16] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:38:05] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:38:05] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:38:13] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:38:16] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.375 second response time [01:38:32] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:38:42] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.871 second response time [01:39:01] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:39:05] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 6.182 second response time [01:39:12] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.11, 6.21, 6.28 [01:40:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 2.70, 5.04, 4.02 [01:41:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.58, 3.37 [01:43:33] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 5.294 second response time [01:44:10] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.791 second response time [01:44:18] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.394 second response time [01:44:24] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.213 second response time [01:44:46] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.668 second response time [01:44:59] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [01:45:54] !log [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php purge (END - exit=0) [01:46:38] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:46:44] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:46:57] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:47:00] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:47:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:47:27] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:48:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.72, 5.65, 4.52 [01:48:40] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.400 second response time [01:48:47] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 6.872 second response time [01:48:51] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.23, 22.73, 21.72 [01:49:06] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.552 second response time [01:49:29] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.148 second response time [01:50:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 2.28, 4.40, 4.19 [01:50:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.98, 21.40, 21.37 [01:51:06] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.168 second response time [01:51:23] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:31] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:52:35] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:52:40] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:53:02] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:53:22] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.30, 6.74, 6.16 [01:53:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.00, 4.30, 3.80 [01:55:10] CosmicAlpha, are you doing WikiSEO next? [01:55:19] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.08, 6.67, 6.22 [01:56:33] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JSnkY [01:56:34] [02miraheze/CreateWiki] 07Universal-Omega 031449030 - Fix style for the reason `selectorother` field [01:56:36] [02CreateWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [01:56:37] [02CreateWiki] 07Universal-Omega opened pull request 03#274: Fix style for the reason `selectorother` field - 13https://git.io/JSnk4 [01:56:52] dmehus: yep doing that now [01:57:06] !log [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php wikiseo (START) [01:57:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:58:07] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 7.02, 7.95, 5.67 [01:58:37] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 0.860 second response time [01:58:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:58:50] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.335 second response time [01:58:55] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:59:12] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.026 second response time [01:59:31] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.008 second response time [01:59:59] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.309 second response time [02:00:32] [02CreateWiki] 07Universal-Omega closed pull request 03#274: Fix style for the reason `selectorother` field - 13https://git.io/JSnk4 [02:00:33] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSnnM [02:00:35] [02miraheze/CreateWiki] 07Universal-Omega 03e1ee9ab - Fix style for the reason `selectorother` field (#274) [02:00:43] [02CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpJTL [02:00:45] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [02:02:01] miraheze/CreateWiki - Universal-Omega the build passed. [02:02:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 2.60, 5.70, 5.30 [02:02:08] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSn8X [02:02:10] [02miraheze/mediawiki] 07Universal-Omega 03d392df5 - Update CreateWiki [02:02:19] CosmicAlpha, cool, thanks :) [02:02:27] dmehus: no problem [02:03:20] !log [universalomega@test3] starting deploy of {'world': True} to skip [02:03:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:04:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 1.90, 4.38, 4.86 [02:04:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:06:29] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.91, 22.16, 19.60 [02:06:33] !log [universalomega@mw11] starting deploy of {'world': True} to all [02:06:51] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.38, 22.16, 21.41 [02:07:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:07:34] miraheze/CreateWiki - Universal-Omega the build passed. [02:07:51] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [02:08:29] !log [universalomega@test3] finished deploy of {'world': True} to skip - SUCCESS in 310s [02:08:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.37, 19.74, 20.62 [02:09:12] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:09:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:10:18] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.46, 22.84, 20.47 [02:10:24] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:10:34] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:10:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.73, 18.67, 20.11 [02:11:06] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:11:18] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:11:20] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:11:27] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:12:23] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.670 second response time [02:12:31] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.664 second response time [02:15:13] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 22161 bytes in 8.905 second response time [02:15:33] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.269 second response time [02:15:45] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.763 second response time [02:16:07] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 10.13, 9.57, 6.72 [02:16:39] !log [universalomega@mw11] finished deploy of {'world': True} to all - SUCCESS in 605s [02:17:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:17:22] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.589 second response time [02:17:38] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.034 second response time [02:19:49] !log [universalomega@mwtask1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php wikiseo (END - exit=0) [02:20:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 3.12, 5.64, 5.71 [02:20:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:21:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.64, 3.96 [02:21:55] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.07, 19.89, 20.28 [02:22:27] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:23:47] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:24:24] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.680 second response time [02:24:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.88, 22.07, 20.99 [02:26:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 2.46, 3.68, 4.81 [02:27:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.91, 3.98 [02:28:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:34:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.39, 5.18, 5.94 [02:34:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.99, 20.03, 20.38 [02:37:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.01, 3.60, 3.87 [02:37:21] [02mediawiki] 07Universal-Omega closed pull request 03#4864: Bump extensions/MirahezeMagic from `b1e2aed` to `8a068e8` - 13https://git.io/JSZ4W [02:37:22] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-8a068e8 - 13https://git.io/vbL5b [02:37:24] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-8a068e8 [02:37:25] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCvS [02:37:27] [02miraheze/mediawiki] 07dependabot[bot] 0371235e9 - Bump extensions/MirahezeMagic from `b1e2aed` to `8a068e8` (#4864) [02:37:29] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCfq [02:37:31] [02miraheze/mediawiki] 07dependabot[bot] 0332ac451 - Bump extensions/Newsletter from `53c9f52` to `3de7955` (#4862) [02:37:32] [02mediawiki] 07Universal-Omega closed pull request 03#4862: Bump extensions/Newsletter from `53c9f52` to `3de7955` - 13https://git.io/JSZ4G [02:37:34] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Newsletter-3de7955 - 13https://git.io/vbL5b [02:37:35] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Newsletter-3de7955 [02:37:38] [02mediawiki] 07Universal-Omega closed pull request 03#4863: Bump extensions/MassEditRegex from `cf4a302` to `27b4608` - 13https://git.io/JSZ4c [02:37:39] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MassEditRegex-27b4608 - 13https://git.io/vbL5b [02:37:41] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MassEditRegex-27b4608 [02:37:42] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCf6 [02:37:44] [02miraheze/mediawiki] 07dependabot[bot] 03708717b - Bump extensions/MassEditRegex from `cf4a302` to `27b4608` (#4863) [02:37:50] [02mediawiki] 07Universal-Omega closed pull request 03#4859: Bump extensions/CheckUser from `db83b3f` to `b2255f3` - 13https://git.io/JSZ4I [02:37:52] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCJq [02:37:53] [02miraheze/mediawiki] 07dependabot[bot] 033d2852c - Bump extensions/CheckUser from `db83b3f` to `b2255f3` (#4859) [02:37:55] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/CheckUser-b2255f3 - 13https://git.io/vbL5b [02:37:56] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/CheckUser-b2255f3 [02:38:02] [02mediawiki] 07Universal-Omega closed pull request 03#4858: Bump extensions/Kartographer from `a2d15e6` to `819f44d` - 13https://git.io/JSZ4J [02:38:03] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCJo [02:38:05] [02miraheze/mediawiki] 07dependabot[bot] 032d3981c - Bump extensions/Kartographer from `a2d15e6` to `819f44d` (#4858) [02:38:06] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Kartographer-819f44d - 13https://git.io/vbL5b [02:38:08] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Kartographer-819f44d [02:38:09] [02mediawiki] 07Universal-Omega closed pull request 03#4855: Bump extensions/Flow from `f7ce11c` to `fd89ee0` - 13https://git.io/JSZ8F [02:38:11] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Flow-fd89ee0 - 13https://git.io/vbL5b [02:38:12] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCJN [02:38:14] [02miraheze/mediawiki] 07dependabot[bot] 03c895a0b - Bump extensions/Flow from `f7ce11c` to `fd89ee0` (#4855) [02:38:15] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Flow-fd89ee0 [02:38:15] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.43, 5.97, 6.07 [02:38:17] [02mediawiki] 07Universal-Omega closed pull request 03#4854: Bump extensions/AntiSpoof from `a309821` to `56b4b17` - 13https://git.io/JSZ8Q [02:38:18] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/AntiSpoof-56b4b17 - 13https://git.io/vbL5b [02:38:20] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/AntiSpoof-56b4b17 [02:38:21] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCUW [02:38:23] [02miraheze/mediawiki] 07dependabot[bot] 034d38aa9 - Bump extensions/AntiSpoof from `a309821` to `56b4b17` (#4854) [02:38:29] [02mediawiki] 07Universal-Omega closed pull request 03#4853: Bump extensions/WikiDiscover from `83eeb61` to `a70309d` - 13https://git.io/JSZ8M [02:38:31] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCUp [02:38:32] [02miraheze/mediawiki] 07dependabot[bot] 03a77d54a - Bump extensions/WikiDiscover from `83eeb61` to `a70309d` (#4853) [02:38:34] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/WikiDiscover-a70309d [02:38:35] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/WikiDiscover-a70309d - 13https://git.io/vbL5b [02:38:45] [02mediawiki] 07Universal-Omega closed pull request 03#4852: Bump extensions/LoginNotify from `3d35729` to `f5e4e43` - 13https://git.io/JSZ8X [02:38:46] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/LoginNotify-f5e4e43 - 13https://git.io/vbL5b [02:38:48] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCTi [02:38:49] [02miraheze/mediawiki] 07dependabot[bot] 039307ca5 - Bump extensions/LoginNotify from `3d35729` to `f5e4e43` (#4852) [02:38:51] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/LoginNotify-f5e4e43 [02:38:52] [02mediawiki] 07Universal-Omega closed pull request 03#4851: Bump extensions/ContactPage from `495144d` to `d5a4827` - 13https://git.io/JSZ86 [02:38:54] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCTj [02:38:55] [02miraheze/mediawiki] 07dependabot[bot] 036395b19 - Bump extensions/ContactPage from `495144d` to `d5a4827` (#4851) [02:38:57] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ContactPage-d5a4827 [02:38:58] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ContactPage-d5a4827 - 13https://git.io/vbL5b [02:39:00] [02mediawiki] 07Universal-Omega closed pull request 03#4850: Bump extensions/PageAssessments from `14d018f` to `851096d` - 13https://git.io/JSZ8V [02:39:01] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PageAssessments-851096d [02:39:03] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PageAssessments-851096d - 13https://git.io/vbL5b [02:39:04] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCkW [02:39:06] [02miraheze/mediawiki] 07dependabot[bot] 03c8737e8 - Bump extensions/PageAssessments from `14d018f` to `851096d` (#4850) [02:39:09] [02mediawiki] 07Universal-Omega closed pull request 03#4849: Bump extensions/AbuseFilter from `b36e6a2` to `4bf6a6a` - 13https://git.io/JSZ82 [02:39:10] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/AbuseFilter-4bf6a6a - 13https://git.io/vbL5b [02:39:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.33, 3.87, 3.94 [02:39:12] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCkD [02:39:13] [02miraheze/mediawiki] 07dependabot[bot] 03dd21909 - Bump extensions/AbuseFilter from `b36e6a2` to `4bf6a6a` (#4849) [02:39:15] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/AbuseFilter-4bf6a6a [02:39:18] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Babel-9e55da4 - 13https://git.io/vbL5b [02:39:19] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCIt [02:39:21] [02miraheze/mediawiki] 07dependabot[bot] 03d0f8bdd - Bump extensions/Babel from `1334838` to `9e55da4` (#4848) [02:39:22] [02mediawiki] 07Universal-Omega closed pull request 03#4848: Bump extensions/Babel from `1334838` to `9e55da4` - 13https://git.io/JSZ8E [02:39:23] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Babel-9e55da4 [02:39:25] [02mediawiki] 07Universal-Omega closed pull request 03#4847: Bump extensions/UserMerge from `57f440b` to `f8306f2` - 13https://git.io/JSZ84 [02:39:27] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCIg [02:39:28] [02miraheze/mediawiki] 07dependabot[bot] 035b5271f - Bump extensions/UserMerge from `57f440b` to `f8306f2` (#4847) [02:39:30] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/UserMerge-f8306f2 [02:39:31] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/UserMerge-f8306f2 - 13https://git.io/vbL5b [02:39:37] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.71, 6.61, 6.14 [02:39:57] [02mediawiki] 07Universal-Omega closed pull request 03#4846: Bump extensions/WikimediaIncubator from `e63ab34` to `12b9543` - 13https://git.io/JSZ8W [02:39:59] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCtt [02:40:00] [02miraheze/mediawiki] 07dependabot[bot] 0358b0d6d - Bump extensions/WikimediaIncubator from `e63ab34` to `12b9543` (#4846) [02:40:02] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/WikimediaIncubator-12b9543 [02:40:03] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/WikimediaIncubator-12b9543 - 13https://git.io/vbL5b [02:40:05] [02mediawiki] 07Universal-Omega closed pull request 03#4845: Bump extensions/TemplateSandbox from `7ebf3bd` to `d7ac520` - 13https://git.io/JSZ8Z [02:40:06] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/TemplateSandbox-d7ac520 [02:40:08] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCtg [02:40:09] [02miraheze/mediawiki] 07dependabot[bot] 037f40d9d - Bump extensions/TemplateSandbox from `7ebf3bd` to `d7ac520` (#4845) [02:40:11] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/TemplateSandbox-d7ac520 - 13https://git.io/vbL5b [02:40:15] [02mediawiki] 07Universal-Omega closed pull request 03#4844: Bump extensions/Score from `bd4416e` to `932fe9e` - 13https://git.io/JSZ8Y [02:40:16] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCtx [02:40:18] [02miraheze/mediawiki] 07dependabot[bot] 034c54bff - Bump extensions/Score from `bd4416e` to `932fe9e` (#4844) [02:40:19] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Score-932fe9e [02:40:21] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Score-932fe9e - 13https://git.io/vbL5b [02:40:27] [02mediawiki] 07Universal-Omega closed pull request 03#4841: Bump extensions/MobileFrontend from `6c4bee5` to `8f64d85` - 13https://git.io/JSZ8f [02:40:28] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCqR [02:40:30] [02miraheze/mediawiki] 07dependabot[bot] 03b15b846 - Bump extensions/MobileFrontend from `6c4bee5` to `8f64d85` (#4841) [02:40:31] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MobileFrontend-8f64d85 [02:40:33] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MobileFrontend-8f64d85 - 13https://git.io/vbL5b [02:40:36] [02mediawiki] 07Universal-Omega closed pull request 03#4840: Bump extensions/UrlShortener from `cb19751` to `820b555` - 13https://git.io/JSZlh [02:40:37] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCqF [02:40:39] [02miraheze/mediawiki] 07dependabot[bot] 034b209b2 - Bump extensions/UrlShortener from `cb19751` to `820b555` (#4840) [02:40:40] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/UrlShortener-820b555 [02:40:42] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/UrlShortener-820b555 - 13https://git.io/vbL5b [02:40:46] [02mediawiki] 07Universal-Omega closed pull request 03#4839: Bump extensions/PagedTiffHandler from `1ca78be` to `5c2fe18` - 13https://git.io/JSZlA [02:40:47] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCmW [02:40:49] [02miraheze/mediawiki] 07dependabot[bot] 03dad8552 - Bump extensions/PagedTiffHandler from `1ca78be` to `5c2fe18` (#4839) [02:40:50] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PagedTiffHandler-5c2fe18 - 13https://git.io/vbL5b [02:40:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.61, 20.85, 20.68 [02:40:52] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PagedTiffHandler-5c2fe18 [02:40:54] [02mediawiki] 07Universal-Omega closed pull request 03#4838: Bump skins/Nimbus from `10aba31` to `7b47d15` - 13https://git.io/JSZl5 [02:40:56] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Nimbus-7b47d15 - 13https://git.io/vbL5b [02:40:57] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Nimbus-7b47d15 [02:40:59] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCmu [02:41:00] [02miraheze/mediawiki] 07dependabot[bot] 0309a4db5 - Bump skins/Nimbus from `10aba31` to `7b47d15` (#4838) [02:41:04] [02mediawiki] 07Universal-Omega closed pull request 03#4837: Bump extensions/TimedMediaHandler from `b2577ee` to `ad5c67f` - 13https://git.io/JSZlQ [02:41:05] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCm2 [02:41:07] [02miraheze/mediawiki] 07dependabot[bot] 03a075eb1 - Bump extensions/TimedMediaHandler from `b2577ee` to `ad5c67f` (#4837) [02:41:08] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/TimedMediaHandler-ad5c67f - 13https://git.io/vbL5b [02:41:10] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/TimedMediaHandler-ad5c67f [02:41:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.99, 3.85, 3.91 [02:41:13] [02mediawiki] 07Universal-Omega closed pull request 03#4836: Bump extensions/PageTriage from `459e8fe` to `5b81b50` - 13https://git.io/JSZlD [02:41:15] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PageTriage-5b81b50 - 13https://git.io/vbL5b [02:41:16] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCmr [02:41:18] [02miraheze/mediawiki] 07dependabot[bot] 0350c4925 - Bump extensions/PageTriage from `459e8fe` to `5b81b50` (#4836) [02:41:20] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PageTriage-5b81b50 [02:41:22] [02mediawiki] 07Universal-Omega closed pull request 03#4835: Bump extensions/Disambiguator from `9b870e9` to `53e8268` - 13https://git.io/JSZlX [02:41:24] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Disambiguator-53e8268 - 13https://git.io/vbL5b [02:41:25] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCm6 [02:41:27] [02miraheze/mediawiki] 07dependabot[bot] 03a53b29b - Bump extensions/Disambiguator from `9b870e9` to `53e8268` (#4835) [02:41:28] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Disambiguator-53e8268 [02:41:32] [02mediawiki] 07Universal-Omega closed pull request 03#4833: Bump extensions/JSBreadCrumbs from `0c9d734` to `0c84de9` - 13https://git.io/JSZlr [02:41:33] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/JSBreadCrumbs-0c84de9 - 13https://git.io/vbL5b [02:41:33] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.22, 6.46, 6.15 [02:41:35] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/JSBreadCrumbs-0c84de9 [02:41:36] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCm1 [02:41:38] [02miraheze/mediawiki] 07dependabot[bot] 03f2406b6 - Bump extensions/JSBreadCrumbs from `0c9d734` to `0c84de9` (#4833) [02:41:47] [02mediawiki] 07Universal-Omega closed pull request 03#4832: Bump extensions/CentralAuth from `da5a133` to `d59e785` - 13https://git.io/JSZla [02:41:49] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/CentralAuth-d59e785 - 13https://git.io/vbL5b [02:41:50] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCmH [02:41:52] [02miraheze/mediawiki] 07dependabot[bot] 03eb772dc - Bump extensions/CentralAuth from `da5a133` to `d59e785` (#4832) [02:41:53] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/CentralAuth-d59e785 [02:42:02] [02mediawiki] 07Universal-Omega closed pull request 03#4829: Bump extensions/DataDump from `bcb4503` to `f47a800` - 13https://git.io/JSZll [02:42:03] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCmp [02:42:05] [02miraheze/mediawiki] 07dependabot[bot] 0357fd5b4 - Bump extensions/DataDump from `bcb4503` to `f47a800` (#4829) [02:42:06] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/DataDump-f47a800 - 13https://git.io/vbL5b [02:42:08] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/DataDump-f47a800 [02:42:11] [02mediawiki] 07Universal-Omega closed pull request 03#4828: Bump skins/MinervaNeue from `75da362` to `7beea49` - 13https://git.io/JSZln [02:42:13] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCYq [02:42:14] [02miraheze/mediawiki] 07dependabot[bot] 03a00b112 - Bump skins/MinervaNeue from `75da362` to `7beea49` (#4828) [02:42:16] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/MinervaNeue-7beea49 [02:42:17] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/MinervaNeue-7beea49 - 13https://git.io/vbL5b [02:42:29] [02mediawiki] 07Universal-Omega closed pull request 03#4827: Bump extensions/AddHTMLMetaAndTitle from `631d970` to `d641e66` - 13https://git.io/JSZlG [02:42:31] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCYo [02:42:32] [02miraheze/mediawiki] 07dependabot[bot] 037d10079 - Bump extensions/AddHTMLMetaAndTitle from `631d970` to `d641e66` (#4827) [02:42:34] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/AddHTMLMetaAndTitle-d641e66 - 13https://git.io/vbL5b [02:42:35] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/AddHTMLMetaAndTitle-d641e66 [02:42:41] [02mediawiki] 07Universal-Omega closed pull request 03#4825: Bump extensions/CentralNotice from `d787a35` to `d3d1795` - 13https://git.io/JSZlL [02:42:42] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCYF [02:42:43] [02miraheze/mediawiki] 07dependabot[bot] 03efa6131 - Bump extensions/CentralNotice from `d787a35` to `d3d1795` (#4825) [02:42:45] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/CentralNotice-d3d1795 - 13https://git.io/vbL5b [02:42:47] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/CentralNotice-d3d1795 [02:42:53] [02mediawiki] 07Universal-Omega closed pull request 03#4824: Bump extensions/MassMessage from `3a0d106` to `bd6de25` - 13https://git.io/JSZlk [02:42:54] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCOq [02:42:56] [02miraheze/mediawiki] 07dependabot[bot] 036917136 - Bump extensions/MassMessage from `3a0d106` to `bd6de25` (#4824) [02:42:57] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MassMessage-bd6de25 [02:42:59] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MassMessage-bd6de25 - 13https://git.io/vbL5b [02:43:02] [02mediawiki] 07Universal-Omega closed pull request 03#4823: Bump extensions/Wikibase from `ba8e322` to `da93956` - 13https://git.io/JSZle [02:43:04] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCOC [02:43:05] [02miraheze/mediawiki] 07dependabot[bot] 03e194c17 - Bump extensions/Wikibase from `ba8e322` to `da93956` (#4823) [02:43:07] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Wikibase-da93956 [02:43:08] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Wikibase-da93956 - 13https://git.io/vbL5b [02:43:12] [02mediawiki] 07Universal-Omega closed pull request 03#4822: Bump extensions/FlaggedRevs from `91e026e` to `796798a` - 13https://git.io/JSZWp [02:43:13] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/FlaggedRevs-796798a - 13https://git.io/vbL5b [02:43:15] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCOl [02:43:16] [02miraheze/mediawiki] 07dependabot[bot] 031a98606 - Bump extensions/FlaggedRevs from `91e026e` to `796798a` (#4822) [02:43:18] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/FlaggedRevs-796798a [02:43:19] [02mediawiki] 07Universal-Omega closed pull request 03#4821: Bump extensions/JsonConfig from `f23fb62` to `20a48c3` - 13https://git.io/JSZWb [02:43:21] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCOE [02:43:22] [02miraheze/mediawiki] 07dependabot[bot] 031b984b7 - Bump extensions/JsonConfig from `f23fb62` to `20a48c3` (#4821) [02:43:24] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/JsonConfig-20a48c3 - 13https://git.io/vbL5b [02:43:25] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/JsonConfig-20a48c3 [02:43:27] [02mediawiki] 07Universal-Omega closed pull request 03#4820: Bump extensions/WikibaseQualityConstraints from `4f9875a` to `9752e3d` - 13https://git.io/JSZW5 [02:43:29] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCOg [02:43:30] [02miraheze/mediawiki] 07dependabot[bot] 0396d007f - Bump extensions/WikibaseQualityConstraints from `4f9875a` to `9752e3d` (#4820) [02:43:32] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/WikibaseQualityConstraints-9752e3d - 13https://git.io/vbL5b [02:43:33] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/WikibaseQualityConstraints-9752e3d [02:43:37] [02mediawiki] 07Universal-Omega closed pull request 03#4818: Bump extensions/CommonsMetadata from `94cf634` to `f06ab37` - 13https://git.io/JSZWM [02:43:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [02:43:38] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCOr [02:43:40] [02miraheze/mediawiki] 07dependabot[bot] 03f3149d5 - Bump extensions/CommonsMetadata from `94cf634` to `f06ab37` (#4818) [02:43:41] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/CommonsMetadata-f06ab37 [02:43:43] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/CommonsMetadata-f06ab37 - 13https://git.io/vbL5b [02:43:49] [02mediawiki] 07Universal-Omega closed pull request 03#4817: Bump extensions/Comments from `f79ada2` to `c390305` - 13https://git.io/JSZWP [02:43:50] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:43:51] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCO1 [02:43:52] [02miraheze/mediawiki] 07dependabot[bot] 032651993 - Bump extensions/Comments from `f79ada2` to `c390305` (#4817) [02:43:54] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Comments-c390305 - 13https://git.io/vbL5b [02:43:55] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Comments-c390305 [02:44:00] [02mediawiki] 07Universal-Omega closed pull request 03#4816: Bump extensions/SocialProfile from `952f5d4` to `f407fdb` - 13https://git.io/JSZWo [02:44:02] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/SocialProfile-f407fdb [02:44:03] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/SocialProfile-f407fdb - 13https://git.io/vbL5b [02:44:05] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCOH [02:44:06] [02miraheze/mediawiki] 07dependabot[bot] 032dc629b - Bump extensions/SocialProfile from `952f5d4` to `f407fdb` (#4816) [02:44:10] [02mediawiki] 07Universal-Omega closed pull request 03#4815: Bump skins/Metrolook from `5dc82cd` to `577b947` - 13https://git.io/JSZWV [02:44:11] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCO5 [02:44:13] [02miraheze/mediawiki] 07dependabot[bot] 03aa6298e - Bump skins/Metrolook from `5dc82cd` to `577b947` (#4815) [02:44:14] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Metrolook-577b947 [02:44:16] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Metrolook-577b947 - 13https://git.io/vbL5b [02:44:21] [02mediawiki] 07Universal-Omega closed pull request 03#4814: Bump extensions/TwoColConflict from `184115a` to `783002e` - 13https://git.io/JSZWg [02:44:23] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/TwoColConflict-783002e [02:44:24] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/TwoColConflict-783002e - 13https://git.io/vbL5b [02:44:25] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCOA [02:44:27] [02miraheze/mediawiki] 07dependabot[bot] 03cc96d40 - Bump extensions/TwoColConflict from `184115a` to `783002e` (#4814) [02:44:31] [02mediawiki] 07Universal-Omega closed pull request 03#4813: Bump extensions/Echo from `3e00e04` to `e2d6620` - 13https://git.io/JSZW0 [02:44:33] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSC3e [02:44:34] [02miraheze/mediawiki] 07dependabot[bot] 03022d7e9 - Bump extensions/Echo from `3e00e04` to `e2d6620` (#4813) [02:44:36] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Echo-e2d6620 - 13https://git.io/vbL5b [02:44:37] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Echo-e2d6620 [02:44:43] [02mediawiki] 07Universal-Omega closed pull request 03#4812: Bump extensions/PollNY from `c13c562` to `1d8efb8` - 13https://git.io/JSZWB [02:44:45] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSC3U [02:44:46] [02miraheze/mediawiki] 07dependabot[bot] 0339db43c - Bump extensions/PollNY from `c13c562` to `1d8efb8` (#4812) [02:44:48] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PollNY-1d8efb8 - 13https://git.io/vbL5b [02:44:49] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PollNY-1d8efb8 [02:44:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.19, 19.87, 20.35 [02:44:52] [02mediawiki] 07Universal-Omega closed pull request 03#4811: Bump extensions/BetaFeatures from `8ce691e` to `a8d6bdb` - 13https://git.io/JSZWc [02:44:53] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/BetaFeatures-a8d6bdb - 13https://git.io/vbL5b [02:44:55] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/BetaFeatures-a8d6bdb [02:44:57] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSC3I [02:44:58] [02miraheze/mediawiki] 07dependabot[bot] 0337bbaff - Bump extensions/BetaFeatures from `8ce691e` to `a8d6bdb` (#4811) [02:45:01] [02mediawiki] 07Universal-Omega closed pull request 03#4810: Bump extensions/Popups from `ac3468d` to `dc1a397` - 13https://git.io/JSZWG [02:45:02] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSC3O [02:45:04] [02miraheze/mediawiki] 07dependabot[bot] 03fcffb20 - Bump extensions/Popups from `ac3468d` to `dc1a397` (#4810) [02:45:05] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Popups-dc1a397 [02:45:07] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Popups-dc1a397 - 13https://git.io/vbL5b [02:45:10] [02mediawiki] 07Universal-Omega closed pull request 03#4809: Bump extensions/WikibaseLexeme from `e65c3d1` to `3901a9b` - 13https://git.io/JSZWm [02:45:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.09, 4.23, 4.03 [02:45:11] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSC3Z [02:45:13] [02miraheze/mediawiki] 07dependabot[bot] 03eecff60 - Bump extensions/WikibaseLexeme from `e65c3d1` to `3901a9b` (#4809) [02:45:14] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/WikibaseLexeme-3901a9b [02:45:16] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/WikibaseLexeme-3901a9b - 13https://git.io/vbL5b [02:45:17] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PageSchemas-7645ba0 - 13https://git.io/vbL5b [02:45:19] [02mediawiki] 07Universal-Omega closed pull request 03#4808: Bump extensions/PageSchemas from `ce663ab` to `7645ba0` - 13https://git.io/JSZWt [02:45:20] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSC3W [02:45:22] [02miraheze/mediawiki] 07dependabot[bot] 03bb8e9b4 - Bump extensions/PageSchemas from `ce663ab` to `7645ba0` (#4808) [02:45:23] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PageSchemas-7645ba0 [02:45:26] [02mediawiki] 07Universal-Omega closed pull request 03#4807: Bump extensions/AuthorProtect from `e1da8e9` to `0741a64` - 13https://git.io/JSZWI [02:45:27] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSC30 [02:45:29] [02miraheze/mediawiki] 07dependabot[bot] 032a71ad2 - Bump extensions/AuthorProtect from `e1da8e9` to `0741a64` (#4807) [02:45:30] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/AuthorProtect-0741a64 [02:45:32] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/AuthorProtect-0741a64 - 13https://git.io/vbL5b [02:45:35] [02mediawiki] 07Universal-Omega closed pull request 03#4806: Bump extensions/SkinPerPage from `9fea664` to `8512bf3` - 13https://git.io/JSZWe [02:45:37] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:45:37] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSC3z [02:45:38] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:45:38] [02miraheze/mediawiki] 07dependabot[bot] 03150aacd - Bump extensions/SkinPerPage from `9fea664` to `8512bf3` (#4806) [02:45:40] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/SkinPerPage-8512bf3 [02:45:41] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/SkinPerPage-8512bf3 - 13https://git.io/vbL5b [02:45:45] [02mediawiki] 07Universal-Omega closed pull request 03#4805: Bump extensions/ManageWiki from `cdda89f` to `a32b21d` - 13https://git.io/JSZCh [02:45:47] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSC3w [02:45:48] [02miraheze/mediawiki] 07dependabot[bot] 034e49a3c - Bump extensions/ManageWiki from `cdda89f` to `a32b21d` (#4805) [02:45:50] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ManageWiki-a32b21d - 13https://git.io/vbL5b [02:45:51] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:45:51] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ManageWiki-a32b21d [02:46:01] [02mediawiki] 07Universal-Omega closed pull request 03#4804: Bump extensions/ProofreadPage from `4c00d71` to `6878509` - 13https://git.io/JSZCb [02:46:02] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ProofreadPage-6878509 [02:46:04] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ProofreadPage-6878509 - 13https://git.io/vbL5b [02:46:05] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSC3M [02:46:07] [02miraheze/mediawiki] 07dependabot[bot] 0340493b8 - Bump extensions/ProofreadPage from `4c00d71` to `6878509` (#4804) [02:46:11] [02mediawiki] 07Universal-Omega closed pull request 03#4803: Bump extensions/GlobalWatchlist from `6417a6d` to `d3cb716` - 13https://git.io/JSZC5 [02:46:13] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSC39 [02:46:14] [02miraheze/mediawiki] 07dependabot[bot] 030697472 - Bump extensions/GlobalWatchlist from `6417a6d` to `d3cb716` (#4803) [02:46:16] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/GlobalWatchlist-d3cb716 [02:46:17] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/GlobalWatchlist-d3cb716 - 13https://git.io/vbL5b [02:46:19] [02mediawiki] 07Universal-Omega closed pull request 03#4802: Bump extensions/wikihiero from `3b35dd6` to `f8c1e38` - 13https://git.io/JSZCQ [02:46:20] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSC37 [02:46:22] [02miraheze/mediawiki] 07dependabot[bot] 032f759af - Bump extensions/wikihiero from `3b35dd6` to `f8c1e38` (#4802) [02:46:23] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/wikihiero-f8c1e38 - 13https://git.io/vbL5b [02:46:25] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/wikihiero-f8c1e38 [02:46:26] [02mediawiki] 07Universal-Omega closed pull request 03#4800: Bump extensions/DynamicPageList from `1ea15de` to `700a600` - 13https://git.io/JSZCX [02:46:28] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSC3b [02:46:29] [02miraheze/mediawiki] 07dependabot[bot] 03ecb8509 - Bump extensions/DynamicPageList from `1ea15de` to `700a600` (#4800) [02:46:31] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/DynamicPageList-700a600 - 13https://git.io/vbL5b [02:46:32] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/DynamicPageList-700a600 [02:46:35] [02mediawiki] 07Universal-Omega closed pull request 03#4799: Bump extensions/ArticlePlaceholder from `bf21a5b` to `d226109` - 13https://git.io/JSZCi [02:46:36] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSC3h [02:46:38] [02miraheze/mediawiki] 07dependabot[bot] 035336be9 - Bump extensions/ArticlePlaceholder from `bf21a5b` to `d226109` (#4799) [02:46:39] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ArticlePlaceholder-d226109 [02:46:41] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ArticlePlaceholder-d226109 - 13https://git.io/vbL5b [02:46:44] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:46:50] [02mediawiki] 07Universal-Omega closed pull request 03#4798: Bump extensions/GlobalUsage from `1ed72d4` to `7071682` - 13https://git.io/JSZCw [02:46:51] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/GlobalUsage-7071682 [02:46:53] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/GlobalUsage-7071682 - 13https://git.io/vbL5b [02:46:54] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCsU [02:46:56] [02miraheze/mediawiki] 07dependabot[bot] 0353dd731 - Bump extensions/GlobalUsage from `1ed72d4` to `7071682` (#4798) [02:47:03] [02mediawiki] 07Universal-Omega closed pull request 03#4797: Bump extensions/GeoData from `ca332d7` to `9a7861c` - 13https://git.io/JSZC2 [02:47:05] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCsq [02:47:06] [02miraheze/mediawiki] 07dependabot[bot] 03eadff10 - Bump extensions/GeoData from `ca332d7` to `9a7861c` (#4797) [02:47:08] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/GeoData-9a7861c - 13https://git.io/vbL5b [02:47:09] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/GeoData-9a7861c [02:47:14] [02mediawiki] 07Universal-Omega closed pull request 03#4796: Bump extensions/UniversalLanguageSelector from `1f6a7af` to `f33d7ce` - 13https://git.io/JSZCE [02:47:16] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-f33d7ce [02:47:17] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCss [02:47:18] [02miraheze/mediawiki] 07dependabot[bot] 0326e43c2 - Bump extensions/UniversalLanguageSelector from `1f6a7af` to `f33d7ce` (#4796) [02:47:20] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-f33d7ce - 13https://git.io/vbL5b [02:47:22] [02mediawiki] 07Universal-Omega closed pull request 03#4795: Bump extensions/AJAXPoll from `1c65c99` to `2ea5085` - 13https://git.io/JSZC8 [02:47:23] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCsn [02:47:25] [02miraheze/mediawiki] 07dependabot[bot] 03cd03145 - Bump extensions/AJAXPoll from `1c65c99` to `2ea5085` (#4795) [02:47:26] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/AJAXPoll-2ea5085 - 13https://git.io/vbL5b [02:47:28] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/AJAXPoll-2ea5085 [02:47:30] [02mediawiki] 07Universal-Omega closed pull request 03#4794: Bump extensions/MsCalendar from `f0501ad` to `d30d8a9` - 13https://git.io/JSZCW [02:47:32] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MsCalendar-d30d8a9 - 13https://git.io/vbL5b [02:47:33] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MsCalendar-d30d8a9 [02:47:35] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCsl [02:47:36] [02miraheze/mediawiki] 07dependabot[bot] 03de89fce - Bump extensions/MsCalendar from `f0501ad` to `d30d8a9` (#4794) [02:47:40] [02mediawiki] 07Universal-Omega closed pull request 03#4793: Bump extensions/Graph from `f687b7d` to `676ba72` - 13https://git.io/JSZCn [02:47:42] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Graph-676ba72 - 13https://git.io/vbL5b [02:47:43] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCsR [02:47:45] [02miraheze/mediawiki] 07dependabot[bot] 03b3b0304 - Bump extensions/Graph from `f687b7d` to `676ba72` (#4793) [02:47:46] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Graph-676ba72 [02:47:50] [02mediawiki] 07Universal-Omega closed pull request 03#4792: Bump extensions/DiscussionTools from `e9efabc` to `7209e5f` - 13https://git.io/JSZCO [02:47:51] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCsg [02:47:53] [02miraheze/mediawiki] 07dependabot[bot] 036b755b0 - Bump extensions/DiscussionTools from `e9efabc` to `7209e5f` (#4792) [02:47:54] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/DiscussionTools-7209e5f [02:47:56] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/DiscussionTools-7209e5f - 13https://git.io/vbL5b [02:48:01] [02mediawiki] 07Universal-Omega closed pull request 03#4791: Bump extensions/ArticleCreationWorkflow from `f7965da` to `d3c2f81` - 13https://git.io/JSZCm [02:48:02] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCsw [02:48:04] [02miraheze/mediawiki] 07dependabot[bot] 036ca42d2 - Bump extensions/ArticleCreationWorkflow from `f7965da` to `d3c2f81` (#4791) [02:48:05] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ArticleCreationWorkflow-d3c2f81 [02:48:07] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ArticleCreationWorkflow-d3c2f81 - 13https://git.io/vbL5b [02:48:10] [02mediawiki] 07Universal-Omega closed pull request 03#4790: Bump extensions/Thanks from `525c1ba` to `06f0c44` - 13https://git.io/JSZCL [02:48:12] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Thanks-06f0c44 - 13https://git.io/vbL5b [02:48:14] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCs6 [02:48:15] [02miraheze/mediawiki] 07dependabot[bot] 03f68f5da - Bump extensions/Thanks from `525c1ba` to `06f0c44` (#4790) [02:48:16] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Thanks-06f0c44 [02:48:33] [02mediawiki] 07Universal-Omega closed pull request 03#4861: Bump extensions/Cargo from `f7ba256` to `07f465d` - 13https://git.io/JSZ43 [02:48:34] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Cargo-07f465d [02:48:36] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCsH [02:48:37] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2607:5300:201:3100::929a/cpweb [02:48:37] [02miraheze/mediawiki] 07dependabot[bot] 039dff05d - Bump extensions/Cargo from `f7ba256` to `07f465d` (#4861) [02:48:39] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Cargo-07f465d - 13https://git.io/vbL5b [02:48:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.65, 21.66, 20.99 [02:49:06] [02mediawiki] 07Universal-Omega closed pull request 03#4860: Bump skins/Femiwiki from `811ebbc` to `21a54f2` - 13https://git.io/JSZ4t [02:49:08] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Femiwiki-21a54f2 - 13https://git.io/vbL5b [02:49:09] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCsj [02:49:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.11, 3.98, 4.00 [02:49:11] [02miraheze/mediawiki] 07dependabot[bot] 037b86757 - Bump skins/Femiwiki from `811ebbc` to `21a54f2` (#4860) [02:49:12] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Femiwiki-21a54f2 [02:49:28] [02mediawiki] 07Universal-Omega closed pull request 03#4857: Bump extensions/HAWelcome from `901f5c0` to `dbc3826` - 13https://git.io/JSZ8h [02:49:30] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/HAWelcome-dbc3826 [02:49:31] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCGI [02:49:33] [02miraheze/mediawiki] 07dependabot[bot] 030d8b118 - Bump extensions/HAWelcome from `901f5c0` to `dbc3826` (#4857) [02:49:34] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/HAWelcome-dbc3826 - 13https://git.io/vbL5b [02:50:17] [02mediawiki] 07Universal-Omega closed pull request 03#4842: Bump extensions/PortableInfobox from `c934b05` to `b0a072f` - 13https://git.io/JSZ8k [02:50:18] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PortableInfobox-b0a072f - 13https://git.io/vbL5b [02:50:20] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCGu [02:50:21] [02miraheze/mediawiki] 07dependabot[bot] 03597be39 - Bump extensions/PortableInfobox from `c934b05` to `b0a072f` (#4842) [02:50:23] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PortableInfobox-b0a072f [02:50:26] [02mediawiki] 07Universal-Omega closed pull request 03#4834: Bump skins/chameleon from `a923c46` to `4428398` - 13https://git.io/JSZl6 [02:50:27] [02mediawiki] 07dependabot[bot] commented on pull request 03#4834: Bump skins/chameleon from `a923c46` to `4428398` - 13https://git.io/JSCGa [02:50:29] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/chameleon-4428398 - 13https://git.io/vbL5b [02:50:30] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/chameleon-4428398 [02:50:51] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.57, 22.76, 21.47 [02:50:54] [02mediawiki] 07Universal-Omega closed pull request 03#4831: Bump extensions/ApprovedRevs from `ef1833a` to `609df70` - 13https://git.io/JSZlE [02:50:56] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCG9 [02:50:57] [02miraheze/mediawiki] 07dependabot[bot] 03c6ff4a8 - Bump extensions/ApprovedRevs from `ef1833a` to `609df70` (#4831) [02:50:59] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ApprovedRevs-609df70 - 13https://git.io/vbL5b [02:51:00] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ApprovedRevs-609df70 [02:51:06] [02mediawiki] 07Universal-Omega closed pull request 03#4830: Bump extensions/PdfHandler from `dcf936c` to `ba526c2` - 13https://git.io/JSZlB [02:51:08] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCGF [02:51:09] [02miraheze/mediawiki] 07dependabot[bot] 0340f435d - Bump extensions/PdfHandler from `dcf936c` to `ba526c2` (#4830) [02:51:11] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PdfHandler-ba526c2 [02:51:12] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PdfHandler-ba526c2 - 13https://git.io/vbL5b [02:51:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:51:21] [02mediawiki] 07Universal-Omega closed pull request 03#4819: Bump skins/Evelution from `da17274` to `3d61d87` - 13https://git.io/JSZWH [02:51:22] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Evelution-3d61d87 - 13https://git.io/vbL5b [02:51:24] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCZv [02:51:25] [02miraheze/mediawiki] 07dependabot[bot] 03ff9bb7e - Bump skins/Evelution from `da17274` to `3d61d87` (#4819) [02:51:27] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Evelution-3d61d87 [02:51:50] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.962 second response time [02:51:53] [02mediawiki] 07Universal-Omega closed pull request 03#4801: Bump skins/Cosmos from `2c46a3c` to `95efb34` - 13https://git.io/JSZCD [02:51:53] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.707 second response time [02:51:54] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Cosmos-95efb34 - 13https://git.io/vbL5b [02:51:56] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Cosmos-95efb34 [02:51:57] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCZZ [02:51:59] [02miraheze/mediawiki] 07dependabot[bot] 0395edb6d - Bump skins/Cosmos from `2c46a3c` to `95efb34` (#4801) [02:52:15] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.014 second response time [02:52:16] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.795 second response time [02:52:16] [02mediawiki] 07Universal-Omega closed pull request 03#4789: Bump extensions/ShortDescription from `543b282` to `d91fac8` - 13https://git.io/JSZCU [02:52:17] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCZ0 [02:52:19] [02miraheze/mediawiki] 07dependabot[bot] 03ebe39e6 - Bump extensions/ShortDescription from `543b282` to `d91fac8` (#4789) [02:52:20] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ShortDescription-d91fac8 - 13https://git.io/vbL5b [02:52:22] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ShortDescription-d91fac8 [02:52:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.19, 21.51, 21.18 [02:52:53] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 22161 bytes in 5.298 second response time [02:53:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.89, 4.40, 4.11 [02:53:36] !log [universalomega@test3] starting deploy of {'world': True, 'l10n': True} to skip [02:54:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:54:07] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 5.27, 6.02, 4.44 [02:54:37] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:56:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 2.41, 4.78, 4.18 [02:56:14] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.93, 6.23, 5.66 [02:58:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.71, 19.28, 20.30 [02:59:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.09, 3.73, 3.91 [03:01:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.14, 4.19, 4.05 [03:02:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.54, 6.31, 5.93 [03:03:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.21, 3.52, 3.83 [03:06:56] !log [universalomega@test3] finished deploy of {'world': True, 'l10n': True} to skip - SUCCESS in 801s [03:07:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:07:48] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True} to all [03:07:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:08:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.47, 5.66, 6.00 [03:08:26] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True} to all [03:08:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:12:25] [02mediawiki] 07Universal-Omega closed pull request 03#4856: Bump skins/Tweeki from `e5f76bc` to `b88f5d7` - 13https://git.io/JSZ8x [03:12:27] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Tweeki-b88f5d7 - 13https://git.io/vbL5b [03:12:28] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCwA [03:12:30] [02miraheze/mediawiki] 07dependabot[bot] 03e852e40 - Bump skins/Tweeki from `e5f76bc` to `b88f5d7` (#4856) [03:12:31] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Tweeki-b88f5d7 [03:12:34] [02mediawiki] 07Universal-Omega closed pull request 03#4843: Bump extensions/PageForms from `1742de7` to `7c19d92` - 13https://git.io/JSZ8q [03:12:35] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PageForms-7c19d92 - 13https://git.io/vbL5b [03:12:37] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCrG [03:12:38] [02miraheze/mediawiki] 07dependabot[bot] 037f32015 - Bump extensions/PageForms from `1742de7` to `7c19d92` (#4843) [03:12:40] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PageForms-7c19d92 [03:12:42] [02mediawiki] 07Universal-Omega closed pull request 03#4826: Bump extensions/Translate from `1c5f696` to `941deb5` - 13https://git.io/JSZlO [03:12:44] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Translate-941deb5 [03:12:45] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSCrw [03:12:47] [02miraheze/mediawiki] 07dependabot[bot] 037f5fbe7 - Bump extensions/Translate from `1c5f696` to `941deb5` (#4826) [03:12:48] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Translate-941deb5 - 13https://git.io/vbL5b [03:12:49] !log [universalomega@test3] starting deploy of {'world': True, 'l10n': True} to skip [03:13:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:16:23] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.99, 20.66, 19.68 [03:17:05] !log [universalomega@test3] finished deploy of {'world': True, 'l10n': True} to skip - SUCCESS in 256s [03:17:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 3.71, 3.67 [03:17:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:18:17] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.42, 19.98, 19.56 [03:18:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [03:19:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.49, 3.60 [03:19:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [03:25:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.75, 3.76, 3.65 [03:25:56] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.85, 20.21, 19.67 [03:26:15] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.72, 5.77, 5.54 [03:27:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.71, 3.65 [03:27:15] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:27:16] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:27:55] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.98, 18.62, 19.15 [03:28:17] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:29:19] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 8.129 second response time [03:29:19] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 22149 bytes in 8.485 second response time [03:29:42] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 31.63, 25.86, 21.82 [03:30:15] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 1.962 second response time [03:30:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:31:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.03, 22.78, 21.20 [03:33:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.00, 4.23, 3.83 [03:33:29] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:34:18] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.41, 5.82, 5.84 [03:35:09] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 1602s [03:35:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:35:55] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.05, 22.12, 20.27 [03:35:57] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True} to all [03:36:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:37:55] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.47, 19.74, 19.64 [03:41:55] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.28, 21.23, 20.29 [03:42:17] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.92, 6.89, 6.15 [03:42:49] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [03:43:26] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [03:43:55] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:43:57] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:44:02] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.80, 5.45, 4.41 [03:45:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:45:52] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.015 second response time [03:45:53] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.315 second response time [03:45:57] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 4.04, 4.87, 4.32 [03:46:03] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 605s [03:46:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:46:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:47:55] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.43, 20.00, 20.15 [03:49:14] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.74, 22.01, 21.30 [03:51:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.79, 3.92 [03:52:14] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.17, 6.58, 5.85 [03:52:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [03:53:08] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.93, 23.10, 21.92 [03:53:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.64, 3.95, 3.95 [03:54:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.41, 5.36, 5.49 [03:54:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:54:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:57:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.15, 3.63, 3.82 [04:06:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.07, 18.60, 20.23 [04:07:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.65, 2.88, 3.39 [04:12:07] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 36.21, 15.44, 7.65 [04:12:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.67, 4.98, 5.99 [04:15:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.48, 4.10, 3.69 [04:15:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 149.56.141.75/cpweb [04:16:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [04:18:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 0.18, 5.10, 5.46 [04:18:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:19:13] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:20:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 0.18, 3.49, 4.82 [04:24:16] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.10, 5.51, 5.47 [04:26:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.61, 5.00, 5.29 [04:29:40] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:33:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.05, 3.74, 3.94 [04:35:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:37:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.77, 3.88 [04:39:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.68, 3.83 [04:40:16] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.59, 4.68, 5.05 [04:40:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:43:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:47:15] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.02, 6.55, 5.82 [04:49:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.54, 3.40, 3.57 [04:49:15] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.63, 6.23, 5.80 [04:51:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.24, 3.35, 3.53 [04:57:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.17, 2.99, 3.35 [05:07:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.31, 3.34 [05:08:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:11:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.65, 3.85, 3.54 [05:13:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:17:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.84, 3.68 [05:19:04] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.57, 6.28, 5.61 [05:21:03] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.56, 5.92, 5.57 [05:21:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.42, 3.78, 3.66 [05:21:37] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 4.78, 7.64, 5.04 [05:22:20] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:22:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [05:23:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.61, 3.61 [05:23:35] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 1.99, 5.63, 4.61 [05:24:15] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.486 second response time [05:24:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:25:34] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 1.46, 4.30, 4.25 [05:28:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:33:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.74, 3.11, 3.39 [05:35:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:35:40] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.69, 5.63, 5.06 [05:37:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.87, 4.07, 3.72 [05:39:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.80, 3.87, 3.68 [05:43:19] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 2.12, 5.24, 4.91 [05:43:28] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.65, 4.77, 4.93 [05:45:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.57, 3.57 [05:45:18] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 7.65, 5.52, 5.01 [05:47:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.89, 3.26, 3.45 [05:47:16] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 2.61, 4.29, 4.61 [05:49:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.53, 2.97, 3.32 [05:55:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.75, 3.56 [05:57:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 3.83, 3.60 [06:03:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.39, 3.66, 3.68 [06:07:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 11.02, 7.37, 4.75 [06:07:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.63, 2.87, 3.32 [06:08:58] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 54090 MB (12% inode=97%); [06:11:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.89, 7.75, 5.51 [06:13:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.35, 8.12, 5.91 [06:25:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.56, 7.56, 7.01 [06:30:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:35:03] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.82, 6.43, 6.70 [06:36:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:44:03] [02ssl] 07Universal-Omega opened pull request 03#463: Remove dreamsit.com.br - 13https://git.io/JSups [06:44:34] [02ssl] 07Universal-Omega synchronize pull request 03#463: Remove dreamsit.com.br - 13https://git.io/JSups [06:46:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:59:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.33, 3.07 [07:00:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:01:53] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.80, 2.99, 2.97 [07:19:09] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.15, 7.58, 6.75 [07:25:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.28, 7.79, 7.17 [07:25:49] .op [07:25:49] CosmicAlpha: Access Denied. If in error, please contact the channel founder. [07:26:19] Yeah didn't think that'd work... [07:33:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.46, 7.17, 7.05 [07:35:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.42, 7.34, 7.14 [07:35:34] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JSzAB [07:35:36] [02miraheze/mw-config] 07Universal-Omega 03bd5ba87 - Remove comments sitenotice [07:35:37] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [07:35:39] [02mw-config] 07Universal-Omega opened pull request 03#4332: Remove comments sitenotice - 13https://git.io/JSzAu [07:36:16] PROBLEM - wiki.cyberfurs.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:36:36] miraheze/mw-config - Universal-Omega the build passed. [07:36:39] [02mw-config] 07Universal-Omega closed pull request 03#4332: Remove comments sitenotice - 13https://git.io/JSzAu [07:36:40] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [07:36:42] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSzx8 [07:36:43] [02miraheze/mw-config] 07Universal-Omega 03c7ce4cc - Remove comments sitenotice (#4332) [07:36:45] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [07:37:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.01, 7.68, 7.27 [07:37:42] miraheze/mw-config - Universal-Omega the build passed. [07:39:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.64, 7.27, 7.17 [07:41:58] !log [@mw11] starting deploy of {'config': True} to ovlon [07:42:16] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 17s [07:42:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:42:25] PROBLEM - wiki.homiedb.party - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:42:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:47:00] !log [@test101] starting deploy of {'config': True} to skip [07:47:01] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [07:47:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:47:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:47:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JSgkH [07:47:20] [02miraheze/mw-config] 07Universal-Omega 0380b61b4 - Change IncidentReporting database for betaheze [07:47:22] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [07:47:34] [02mw-config] 07Universal-Omega opened pull request 03#4333: Change IncidentReporting database for betaheze - 13https://git.io/JSgIf [07:47:43] [02puppet] 07Universal-Omega opened pull request 03#2248: Add grant for testincidents database - 13https://git.io/JSgIY [07:48:34] [02mw-config] 07Universal-Omega edited pull request 03#4333: Change IncidentReporting database for betaheze - 13https://git.io/JSgIf [07:48:39] miraheze/mw-config - Universal-Omega the build passed. [07:49:13] [02puppet] 07Universal-Omega edited pull request 03#2248: Add grant for testincidents database - 13https://git.io/JSgIY [07:50:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JSgtj [07:50:36] [02miraheze/mw-config] 07Universal-Omega 035f2e4fd - Update LocalSettings.php [07:50:38] [02mw-config] 07Universal-Omega synchronize pull request 03#4333: Change IncidentReporting database for betaheze - 13https://git.io/JSgIf [07:50:39] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [07:51:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [07:51:40] miraheze/mw-config - Universal-Omega the build passed. [07:51:45] !log [@mwtask111] starting deploy of {'config': True} to scsvg [07:51:46] !log [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost [07:52:20] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:52:33] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:52:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:53:03] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.79, 6.30, 6.79 [07:53:46] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:53:51] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:53:55] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:54:09] !log [@test3] starting deploy of {'config': True} to skip [07:54:10] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 1s [07:55:10] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [07:55:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:55:28] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [07:56:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:56:27] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.795 second response time [07:56:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:56:45] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.396 second response time [07:57:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.37, 6.81, 6.88 [07:57:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:57:33] RECOVERY - wiki.cyberfurs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cyberfurs.org' will expire on Tue 29 Mar 2022 06:21:09 GMT +0000. [07:57:49] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 22161 bytes in 0.301 second response time [07:57:49] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.324 second response time [07:58:05] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.445 second response time [07:59:09] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:03:04] RECOVERY - wiki.homiedb.party - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.homiedb.party' will expire on Sun 27 Mar 2022 23:45:33 GMT +0000. [08:05:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.18, 7.56, 7.19 [08:07:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.88, 7.41, 7.17 [08:15:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.52, 7.53, 7.27 [08:17:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.76, 7.53, 7.30 [08:18:37] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:19:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.84, 8.15, 7.54 [08:21:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.38, 7.38, 7.33 [08:23:26] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:24:31] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.85, 1.60, 0.98 [08:26:31] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.86, 1.28, 0.94 [08:31:03] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 4.95, 6.07, 6.76 [08:37:55] CosmicAlpha, have you been able to op with MirahezeBot since you changed your nick? [08:38:12] I might just need to do a MirahezeBots PR and merge it for you [08:45:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.18, 6.76, 6.70 [08:45:33] dmehus: I just changed the ACCOUNTNAME today. [08:46:43] CosmicAlpha: oh, didn't realize that; I thought it went by nick, not logged in username. Nevertheless, unfortunately, I can only merge to certain MirahezeBots repos (not Puppet) as I only have shell access to `bots1` [08:46:58] so needs MacFan4000, RhinosF1, or Sarip [08:47:04] s/Sarip/Sario [08:47:04] dmehus meant to say: so needs MacFan4000, RhinosF1, or Sario [08:47:36] dmehus: yeah, it goes by account name. [08:49:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.65, 7.40, 6.95 [08:49:12] CosmicAlpha, ah [08:49:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.45, 3.02 [08:51:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.78, 3.18, 2.98 [08:57:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 2.30, 7.50, 7.97 [08:57:51] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JS23N [08:57:53] [02miraheze/mw-config] 07Universal-Omega 031d2609f - Add sitenotice about SCSVG and recent issues [08:57:54] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [08:57:56] [02mw-config] 07Universal-Omega opened pull request 03#4334: Add sitenotice about SCSVG and recent issues - 13https://git.io/JS23h [08:58:11] ssh-agent: ^ [08:58:18] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JS2sB [08:58:20] [02miraheze/mw-config] 07github-actions 038779fdf - CI: lint code to MediaWiki standards [08:58:21] [02mw-config] 07github-actions[bot] synchronize pull request 03#4334: Add sitenotice about SCSVG and recent issues - 13https://git.io/JS23h [08:58:39] CosmicAlpha: looks good, +1 [08:58:56] miraheze/mw-config - Universal-Omega the build passed. [09:00:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:01:03] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 2.48, 4.79, 6.75 [09:05:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JS2lP [09:05:16] [02miraheze/mw-config] 07Universal-Omega 0390a54ba - Grammar [09:05:18] [02mw-config] 07Universal-Omega synchronize pull request 03#4334: Add sitenotice about SCSVG and recent issues - 13https://git.io/JS23h [09:06:15] miraheze/mw-config - Universal-Omega the build passed. [09:07:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JS24S [09:07:24] [02miraheze/mw-config] 07Universal-Omega 03a232d9d - Update Sitenotice.php [09:07:25] [02mw-config] 07Universal-Omega synchronize pull request 03#4334: Add sitenotice about SCSVG and recent issues - 13https://git.io/JS23h [09:08:21] miraheze/mw-config - Universal-Omega the build passed. [09:08:41] morning RhinosF1 [09:08:51] I almost said morning in -cvt-feed heh [09:09:03] Morning dmehus [09:18:26] [02mw-config] 07Universal-Omega closed pull request 03#4334: Add sitenotice about SCSVG and recent issues - 13https://git.io/JS23h [09:18:28] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [09:18:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JS2rn [09:18:31] [02miraheze/mw-config] 07Universal-Omega 03bb07b7f - Add sitenotice about SCSVG and recent issues (#4334) [09:18:32] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [09:19:32] miraheze/mw-config - Universal-Omega the build passed. [09:21:41] !log [@mwtask111] starting deploy of {'config': True} to scsvg [09:21:42] !log [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost [09:21:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:21:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:23:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:23:59] ok night [09:24:41] !log [@test3] starting deploy of {'config': True} to skip [09:24:42] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [09:24:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:24:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:25:28] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [09:28:07] Well done [09:41:40] !log [@mw11] starting deploy of {'config': True} to ovlon [09:41:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:42:02] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 22s [09:42:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:45:29] .op [09:45:29] CosmicAlpha: Access Denied. If in error, please contact the channel founder. [09:46:00] RhinosF1: ^ does something else need done to deploy that MirahezeBots PR? [09:46:26] .checkchanopcache [09:46:26] RhinosF1: Cache does not match on-disk copy [09:46:31] .resetchanopcache [09:46:31] RhinosF1: Refreshing Cache... [09:46:32] RhinosF1: Cache refreshed [09:46:34] Done CosmicAlpha [09:46:50] Thanks! [09:46:52] .op [09:46:52] Attempting to OP... [09:47:03] .deop [09:47:03] Attempting to OP... [09:47:47] !log [@test101] starting deploy of {'config': True} to skip [09:47:48] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [09:47:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:47:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:50:39] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [09:53:26] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:58:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:00:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.04, 5.77, 4.77 [10:02:24] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.12, 5.06, 4.62 [10:03:54] [02mw-config] 07AgentIsai opened pull request 03#4335: Add link to CN post regarding performance issues - 13https://git.io/JSaCH [10:04:30] ssh-agent: need a merge? [10:04:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:04:57] miraheze/mw-config - AgentIsai the build passed. [10:05:00] RhinosF1: Yes please [10:05:18] [02mw-config] 07RhinosF1 closed pull request 03#4335: Add link to CN post regarding performance issues - 13https://git.io/JSaCH [10:05:19] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSalc [10:05:20] [02miraheze/mw-config] 07AgentIsai 0327a3e04 - Add link to CN post regarding performance issues (#4335) [10:05:25] Thanks :) [10:05:28] There you go [10:05:53] Standard disclaimer that it will deploy within 30 minutes to all 4 stages [10:06:02] And cause at least 2 icinga alerts [10:06:30] alright, thanks [10:06:31] miraheze/mw-config - RhinosF1 the build passed. [10:09:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:11:35] !log [@mw11] starting deploy of {'config': True} to ovlon [10:11:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:11:42] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 6s [10:11:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:14:21] ssh-agent: now deployed to active production [10:15:14] RhinosF1: exciting, do we have a timeline yet of when MW traffic will be moved over to the new stack? [10:15:37] ssh-agent: mediawiki is waiting for traffic [10:15:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:15:48] It's entirely infra's work now [10:15:53] ah, alright [10:16:05] There's nothing on our team we can do [10:16:28] Beyond deploy i18n when we get it & make sure caches are there [10:16:35] But we need a DB first [10:17:15] Cool, sounds good [10:17:30] !log [@test101] starting deploy of {'config': True} to skip [10:17:31] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [10:17:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:17:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:19:56] ssh-agent: the deploy aborteds are expected [10:20:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:21:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.25, 2.90 [10:21:37] !log [@mwtask111] starting deploy of {'config': True} to scsvg [10:21:38] !log [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost [10:21:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:22:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:23:15] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.21, 6.60, 5.37 [10:24:13] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.94, 6.03, 5.32 [10:24:31] !log [@test3] starting deploy of {'config': True} to skip [10:24:32] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [10:24:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:24:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:24:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:25:15] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.95, 6.16, 5.36 [10:25:27] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [10:26:12] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.21, 5.52, 5.23 [10:27:43] !log downtime mwtask111 and test101 puppet alerts as they are flappy and annoying [10:27:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:29:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.12, 3.02, 3.01 [10:36:57] !log updated MW grafana-new to fix php alerts [10:37:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:39:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:44:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:45:51] !log deploy world+config to SCSVG with --force to fix [10:45:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:49:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:54:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:04:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:17:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:25:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.35, 3.13 [11:27:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.10, 3.26, 3.13 [11:31:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.66, 3.31 [11:33:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.94, 3.34, 3.23 [11:42:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:46:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:47:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.11, 3.41, 3.09 [11:49:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.27, 3.01, 2.99 [11:50:50] PROBLEM - db11 Disk Space on db11 is WARNING: DISK WARNING - free space: / 39369 MB (8% inode=97%); [12:06:33] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JSVA7 [12:06:34] [02miraheze/landing] 07translatewiki 03ab2e3a4 - Localisation updates from https://translatewiki.net. [12:06:35] [url] Main page - translatewiki.net | translatewiki.net [12:06:36] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JSVA5 [12:06:37] [02miraheze/CreateWiki] 07translatewiki 034e40b5a - Localisation updates from https://translatewiki.net. [12:06:38] [url] Main page - translatewiki.net | translatewiki.net [12:06:39] [02miraheze/MatomoAnalytics] 07translatewiki pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JSVAd [12:06:40] [02miraheze/MatomoAnalytics] 07translatewiki 03ba5a4c8 - Localisation updates from https://translatewiki.net. [12:06:41] [url] Main page - translatewiki.net | translatewiki.net [12:06:42] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JSVAF [12:06:43] [02miraheze/MirahezeMagic] 07translatewiki 03a9b4b27 - Localisation updates from https://translatewiki.net. [12:06:44] [url] Main page - translatewiki.net | translatewiki.net [12:09:23] miraheze/landing - translatewiki the build passed. [12:16:23] miraheze/CreateWiki - translatewiki the build passed. [12:16:30] miraheze/MatomoAnalytics - translatewiki the build passed. [12:16:39] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JSwTM [12:16:40] [02miraheze/puppet] 07JohnFLewis 03576163a - db: increase bin log duration + disable automatic backups [12:17:00] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [12:17:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:18:43] miraheze/MirahezeMagic - translatewiki the build passed. [12:21:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [12:21:44] !log [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost [12:21:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:22:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:24:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.32, 3.33, 2.90 [12:25:55] !log redoing db backups with schemas, disabled automatic backups and increased binlogs so hopefully reduce long term db load for now [12:26:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:26:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.34, 2.95 [12:27:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.72, 5.83, 4.14 [12:28:16] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.39, 3.28, 2.96 [12:28:21] PROBLEM - db13 Current Load on db13 is WARNING: WARNING - load average: 7.63, 4.91, 2.99 [12:30:21] PROBLEM - db13 Current Load on db13 is CRITICAL: CRITICAL - load average: 8.16, 5.98, 3.62 [12:32:21] PROBLEM - db13 Current Load on db13 is WARNING: WARNING - load average: 7.17, 6.45, 4.09 [12:36:21] PROBLEM - db13 Current Load on db13 is CRITICAL: CRITICAL - load average: 8.96, 7.70, 5.13 [12:38:21] PROBLEM - db13 Current Load on db13 is WARNING: WARNING - load average: 7.44, 7.46, 5.34 [12:44:29] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 52527 MB (11% inode=97%); [12:44:58] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 47600 MB (10% inode=97%); [12:50:21] PROBLEM - db13 Current Load on db13 is CRITICAL: CRITICAL - load average: 9.45, 8.10, 6.72 [12:52:21] PROBLEM - db13 Current Load on db13 is WARNING: WARNING - load average: 7.17, 7.53, 6.67 [12:53:23] PROBLEM - bacula2 Bacula Databases db13 on bacula2 is CRITICAL: CRITICAL: Timeout or unknown client: db13-fd [12:53:59] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CRITICAL: Timeout or unknown client: puppet3-fd [12:54:29] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48182 MB (10% inode=97%); [12:54:58] PROBLEM - bacula2 Bacula Databases db11 on bacula2 is CRITICAL: CRITICAL: Timeout or unknown client: db11-fd [12:55:00] PROBLEM - bacula2 Bacula Databases db12 on bacula2 is CRITICAL: CRITICAL: Timeout or unknown client: db12-fd [12:57:21] PROBLEM - bacula2 Bacula Static on bacula2 is CRITICAL: CRITICAL: Timeout or unknown client: gluster3-fd [12:57:46] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CRITICAL: Timeout or unknown client: phab2-fd [12:58:21] RECOVERY - db13 Current Load on db13 is OK: OK - load average: 5.40, 6.35, 6.44 [13:06:40] [02dns] 07RhinosF1 synchronize pull request 03#233: mediawiki: failover deployment & mwtask - 13https://git.io/JDyu6 [13:09:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.60, 6.98, 7.89 [13:15:36] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.92, 6.04, 4.73 [13:15:58] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:16:09] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:16:20] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:17:26] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [13:17:57] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.159 second response time [13:18:10] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.901 second response time [13:18:20] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.941 second response time [13:18:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [13:19:35] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.40, 6.61, 5.33 [13:19:43] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:19:48] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:20] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:21:11] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:21:26] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:21:42] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.438 second response time [13:21:43] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 1.767 second response time [13:22:20] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.772 second response time [13:22:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:23:13] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.675 second response time [13:44:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb [13:47:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.37, 7.25, 7.09 [13:49:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.26, 7.48, 7.21 [13:51:36] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:52:15] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [13:53:31] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.962 second response time [13:53:53] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:59] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:54:09] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:54:34] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:54:40] PROBLEM - phab2 Current Load on phab2 is CRITICAL: CRITICAL - load average: 5.24, 4.32, 1.94 [13:54:55] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:55:54] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 6.560 second response time [13:56:00] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.487 second response time [13:56:13] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.817 second response time [13:59:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.84, 7.36, 7.07 [14:00:13] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:00:40] RECOVERY - phab2 Current Load on phab2 is OK: OK - load average: 0.32, 1.50, 1.41 [14:00:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:00:52] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.704 second response time [14:01:11] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.329 second response time [14:09:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.22, 7.50, 7.43 [14:12:31] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.41, 1.71, 1.19 [14:13:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.97, 7.72, 7.49 [14:14:31] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.20, 1.52, 1.18 [14:22:11] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [14:24:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:29:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.86, 7.74, 7.84 [14:31:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.11, 8.27, 8.02 [14:33:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 5.89, 7.42, 7.74 [14:35:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 11.24, 8.58, 8.11 [14:39:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.11, 7.62, 7.82 [14:40:30] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 52461 MB (11% inode=97%); [14:48:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [14:53:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.21, 7.26, 7.35 [14:54:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:55:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.35, 7.11, 7.28 [14:55:23] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.46, 3.05 [14:57:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.15, 7.52, 7.40 [14:57:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.65, 3.08, 2.95 [15:08:29] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48574 MB (10% inode=97%); [15:09:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.67, 8.00, 7.96 [15:10:31] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 9.96, 5.06, 2.53 [15:11:19] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.46, 1.72, 1.31 [15:13:19] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.68, 1.35, 1.22 [15:15:55] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [15:16:29] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 49267 MB (11% inode=97%); [15:17:54] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:19:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.68, 7.95, 7.78 [15:19:51] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [15:20:32] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.79, 1.52, 1.84 [15:21:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.32, 7.44, 7.63 [15:21:45] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:22:29] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48563 MB (10% inode=97%); [15:23:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.16, 7.38, 7.56 [15:26:31] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.06, 1.23, 1.59 [15:29:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.22, 7.43, 7.53 [15:38:39] PROBLEM - db13 Current Load on db13 is WARNING: WARNING - load average: 6.09, 6.81, 6.42 [15:39:03] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.61, 6.11, 6.77 [15:39:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.73, 3.79, 3.30 [15:42:33] RECOVERY - db13 Current Load on db13 is OK: OK - load average: 5.21, 6.31, 6.34 [15:43:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.07, 7.26, 7.05 [15:43:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.73, 3.62, 3.34 [15:43:45] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:45:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.66, 7.16, 7.06 [15:45:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.72, 3.27, 3.24 [15:45:45] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:46:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:47:28] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [15:49:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.60, 8.14, 7.44 [15:49:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.47, 3.51, 3.34 [15:51:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.51, 7.87, 7.43 [15:51:36] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:51:42] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb [15:51:58] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:53:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.24, 8.16, 7.59 [15:53:11] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:53:31] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.046 second response time [15:53:41] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:53:56] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 3.082 second response time [15:55:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.43, 7.55, 7.44 [15:57:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.73, 3.97, 3.60 [15:59:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.61, 3.50 [16:01:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.23, 7.73, 7.50 [16:01:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.86, 3.31, 3.40 [16:14:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::5ebc/cpweb [16:16:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:18:34] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.15, 1.71, 1.21 [16:20:31] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.89, 1.39, 1.15 [16:23:53] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.57, 3.31 [16:25:50] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.91, 3.40, 3.28 [16:33:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.34, 7.29, 7.87 [16:36:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.33, 3.20 [16:39:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.88, 7.96, 7.91 [16:40:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.65, 3.33 [16:40:50] PROBLEM - db11 Disk Space on db11 is CRITICAL: DISK CRITICAL - free space: / 26652 MB (5% inode=97%); [16:41:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.59, 7.85, 7.88 [16:42:22] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.17, 3.14, 3.19 [16:45:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.84, 8.60, 8.16 [16:48:00] [02puppet] 07Universal-Omega opened pull request 03#2249: grafana-webhook: support IPV6 - 13https://git.io/JS6RD [16:49:54] paladox: ^ [16:50:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.93, 3.86, 3.42 [16:50:12] [02puppet] 07paladox closed pull request 03#2249: grafana-webhook: support IPV6 - 13https://git.io/JS6RD [16:50:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JS6Ev [16:50:15] [02miraheze/puppet] 07Universal-Omega 0365c7607 - grafana-webhook: support IPV6 (#2249) [16:51:32] Thanks! [16:52:06] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.79, 3.64, 3.38 [16:52:42] CosmicAlpha: I fixed the alerts showing 100% [16:52:49] They now use active+idle [16:53:00] Instead of active+max_active [16:56:31] RhinosF1: nice thanks! [16:57:58] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.76, 3.28, 3.32 [17:01:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.68, 7.45, 7.95 [17:13:07] [02puppet] 07Universal-Omega opened pull request 03#2250: irc: use irc.libera.chat rather than direct IPv6 address - 13https://git.io/JS62b [17:13:49] [02puppet] 07paladox commented on pull request 03#2250: irc: use irc.libera.chat rather than direct IPv6 address - 13https://git.io/JS62x [17:14:16] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [17:14:38] [02puppet] 07Universal-Omega commented on pull request 03#2250: irc: use irc.libera.chat rather than direct IPv6 address - 13https://git.io/JS6aU [17:15:03] [02puppet] 07dmehus commented on pull request 03#2250: irc: use irc.libera.chat rather than direct IPv6 address - 13https://git.io/JS6aL [17:15:53] [02puppet] 07Universal-Omega commented on pull request 03#2250: irc: use irc.libera.chat rather than direct IPv6 address - 13https://git.io/JS6aG [17:16:15] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:17:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.05, 7.13, 7.34 [17:21:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.09, 7.07, 7.27 [17:29:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.23, 7.04, 7.08 [17:31:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.34, 6.93, 7.02 [17:33:03] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.17, 6.31, 6.79 [17:39:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.13, 6.93, 6.92 [17:41:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.68, 7.46, 7.10 [17:43:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.10, 7.15, 7.05 [17:45:24] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:47:03] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.86, 7.67, 7.24 [17:47:24] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.045 second response time [17:50:52] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:52:11] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [17:52:18] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:52:59] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.707 second response time [17:53:03] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.26, 7.34, 7.25 [17:56:45] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:56:48] [02puppet] 07Universal-Omega opened pull request 03#2251: stunnel: fix IPv6 IPs + correct mwtask111 - 13https://git.io/JS662 [17:56:49] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:57:05] paladox: ^ [17:57:17] i'm not sure that's correct. [17:57:24] CosmicAlpha: if you are going on icinga [17:57:26] Don't [17:57:44] They fail because it's based on where the cp is [17:57:50] And cp* still use old puppet [17:58:00] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:58:02] So are running of the old monitoring [17:58:07] Which has no access [17:58:11] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:58:38] paladox: that should be correct if it's using the port, I think? And also mwtask111 was using mwtask1 IPv6. I didn't get that from icinga RhinosF1, just noticed in puppet. [17:58:39] grafana-new works [17:58:45] and that's using cache proxy [17:58:47] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.170 second response time [17:58:52] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.936 second response time [17:59:07] CosmicAlpha: ok with the IP but are you sure the syntax is right [17:59:18] paladox: does the monitoring check [17:59:49] Mon does [17:59:52] All IPv6 are supposed to be enclosed in brackets when a port is given or it takes the port as part of the IP usually. Or at least from my understanding. [17:59:58] But none of the others do [18:00:14] CosmicAlpha: doesn't mean everyone implements it that way [18:01:15] firewall isn't open to it because it doesn't share the same puppet server (cp*) [18:02:43] CosmicAlpha: can you please try and test it [18:03:03] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.69, 6.36, 6.78 [18:03:22] RhinosF1: not sure how I would test it? [18:03:51] Hang on no [18:03:59] We can't until cp* move puppets [18:04:41] You'd have to check docs [18:04:56] I didn't see anything when I wrote it regarding format [18:05:43] curl localhost:8300 [18:05:45] works [18:06:19] curl https://mw101.miraheze.org:443 just hangs? [18:06:22] oh [18:06:25] i know why [18:06:32] firewall not open to it [18:06:49] yeah [18:07:17] We can't test without an open firewall [18:07:49] paladox: what does the localhost:8300 actually give [18:08:51] the contents of grafana [18:09:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.44, 4.07, 3.46 [18:09:26] I'll just update PR to fix mwtask111 I guess. [18:09:30] paladox: so the format must be right [18:09:32] CosmicAlpha: yes [18:09:42] yeh the format is correct [18:10:36] [02puppet] 07Universal-Omega synchronize pull request 03#2251: stunnel: fix IPv6 IPs + correct mwtask111 - 13https://git.io/JS662 [18:10:45] [02puppet] 07Universal-Omega edited pull request 03#2251: stunnel: fix mwtask111 IP - 13https://git.io/JS662 [18:10:56] [02puppet] 07paladox closed pull request 03#2251: stunnel: fix mwtask111 IP - 13https://git.io/JS662 [18:10:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JS6PQ [18:10:59] [02miraheze/puppet] 07Universal-Omega 03ee35f0a - stunnel: fix mwtask111 IP (#2251) [18:11:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [18:11:49] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:11:55] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:12:02] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:12:08] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:19] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.42, 1.84, 1.14 [18:15:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.69, 3.72, 3.55 [18:15:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:15:19] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.57, 1.38, 1.05 [18:15:54] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.021 second response time [18:15:56] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.343 second response time [18:16:10] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.019 second response time [18:16:11] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.358 second response time [18:21:50] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:22:58] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.76, 1.23, 0.92 [18:24:59] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.53, 1.35, 1.00 [18:25:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.43, 3.44 [18:27:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.36, 3.42 [18:29:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.63, 3.12, 3.32 [18:30:49] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:30:57] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:31:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:31:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:31:21] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:32:50] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.932 second response time [18:32:56] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.604 second response time [18:33:08] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:33:18] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.890 second response time [18:38:59] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [18:39:48] will the cps be one of the next to switch? [18:44:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.96, 6.97, 6.12 [18:44:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:46:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.27, 6.14, 5.90 [18:48:40] switch? [18:54:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 3.63, 3.33 [18:55:15] JohnLewis, yeah what I mean is when end users will be routed to the cps on the new infra rather than the cps on the OVH infra [18:55:35] The cps in OVH are ‘new infra’ [18:55:45] We’re keeping them external [18:55:50] We're not migrating them to SC? [18:55:52] ah! [18:56:01] so we will still have an OVH bill [18:56:09] it'll just be a much smaller OVH bill :) [18:56:19] We will have one anyway as we’re moving backups to OVH instead of RamNode [18:56:31] ah [18:56:46] and 1&1 will still be for the nameservers? [18:56:52] Yeah [18:56:55] ah [18:57:24] I guess I was confused where RhinosF1 said "We can't until cp* move puppets" [18:57:43] but I guess he meant the existing cps being controlled by the new puppet server [18:57:52] so I misread that [18:58:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.71, 3.45 [18:59:31] actually, keeping the cps external to SCSVG makes sense and answers a question I had...how we'd still manage to have cps in different geographical locations :P [19:01:28] Yes [19:01:35] We need to move puppet [19:02:43] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.51, 3.62, 3.43 [19:02:48] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.70, 5.51, 3.68 [19:04:43] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.40, 4.85, 3.66 [19:06:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.84, 3.57 [19:11:07] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 5.17, 6.86, 4.28 [19:12:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.06, 3.17, 3.36 [19:13:07] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 3.01, 5.49, 4.09 [19:15:07] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 2.80, 4.59, 3.93 [19:17:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [19:17:57] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [19:18:54] RhinosF1, ack, thanks :) [19:20:16] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.40, 4.23, 3.11 [19:20:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.58, 18.74, 16.80 [19:21:07] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 3.55, 5.48, 4.65 [19:21:32] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:21:48] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:21:53] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:21:53] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.31, 7.15, 6.03 [19:22:07] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:22:16] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.05, 4.10, 3.20 [19:22:20] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:22:35] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:22:50] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:22:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.14, 19.37, 17.30 [19:23:00] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:07] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 2.44, 4.42, 4.36 [19:23:31] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:23:37] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.103 second response time [19:23:44] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 2.542 second response time [19:23:50] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.29, 7.17, 6.17 [19:23:56] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.331 second response time [19:24:06] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.851 second response time [19:24:21] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.005 second response time [19:24:31] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.93, 1.66, 1.13 [19:24:55] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.318 second response time [19:25:47] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.00, 6.48, 6.05 [19:26:31] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.46, 1.68, 1.21 [19:26:55] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.731 second response time [19:27:08] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 7.030 second response time [19:27:13] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:27:39] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.118 second response time [19:27:53] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:28:11] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-3 [+1/-0/±0] 13https://git.io/JSivM [19:28:12] [02miraheze/dns] 07paladox 035a57849 - Add ptr record for mon111 [19:28:14] [02dns] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbQXl [19:28:15] [02dns] 07paladox opened pull request 03#240: Add ptr record for mon111 - 13https://git.io/JSivy [19:30:30] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JSifz [19:30:31] [02miraheze/dns] 07paladox 0393438d0 - Update 0.0.0.0.0.4.7.6.0.1.a.2.ip6.arpra [19:30:33] [02dns] 07paladox synchronize pull request 03#240: Add ptr record for mon111 - 13https://git.io/JSivy [19:31:08] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 3.55, 7.15, 5.94 [19:32:36] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JSiJL [19:32:38] [02miraheze/dns] 07paladox 03e40fa4a - Update 0.0.0.0.0.4.7.6.0.1.a.2.ip6.arpra [19:32:39] [02dns] 07paladox synchronize pull request 03#240: Add ptr record for mon111 - 13https://git.io/JSivy [19:33:21] [02dns] 07paladox closed pull request 03#240: Add ptr record for mon111 - 13https://git.io/JSivy [19:33:22] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JSiJc [19:33:24] [02miraheze/dns] 07paladox 034980b65 - Add ptr record for mon111 (#240) [19:37:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.59, 3.34 [19:39:08] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 3.38, 5.18, 5.60 [19:39:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.25, 3.31, 3.26 [19:41:07] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 6.35, 6.38, 6.03 [19:46:40] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:47:05] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.28, 2.31, 1.51 [19:47:07] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 2.41, 5.28, 5.98 [19:49:03] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.82, 1.76, 1.41 [19:49:30] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [19:49:36] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:49:57] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:49:58] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:50:14] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:48] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:51:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.78, 1.49, 1.35 [19:51:15] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:53:07] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 2.36, 3.47, 4.94 [19:53:15] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.301 second response time [19:53:18] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:53:35] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.768 second response time [19:54:11] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 7.732 second response time [19:54:15] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.723 second response time [19:54:23] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 6.993 second response time [19:54:46] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:58:16] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:59:13] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.006 second response time [19:59:48] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:59:48] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.006 second response time [19:59:58] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:01:07] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.008 second response time [20:01:44] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:01:45] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:01:50] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:01:56] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:01:58] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:02:02] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:02:02] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:02:11] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:02:24] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.249 second response time [20:02:31] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.273 second response time [20:02:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:03:41] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.378 second response time [20:03:44] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 5.237 second response time [20:03:47] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.878 second response time [20:03:48] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.125 second response time [20:03:55] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.462 second response time [20:04:00] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.158 second response time [20:04:02] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.366 second response time [20:04:07] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 7.633 second response time [20:04:08] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.275 second response time [20:04:11] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23663 bytes in 5.532 second response time [20:04:23] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.323 second response time [20:04:25] RhinosF1, am I still okay to do global renames in the next few hours? [20:04:31] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.159 second response time [20:04:34] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.439 second response time [20:04:47] dmehus: you will get a weeks notice [20:04:50] So yes [20:04:55] RhinosF1, ack, SGTM [20:04:57] thanks :) [20:07:37] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:07:41] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:07:45] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:08:15] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:08:58] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:08:59] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:09:01] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:09:11] PROBLEM - mail2 webmail.miraheze.org HTTPS on mail2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:09:35] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.232 second response time [20:09:38] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 1.111 second response time [20:09:41] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.338 second response time [20:10:14] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.791 second response time [20:10:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:10:59] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 2.870 second response time [20:11:00] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.646 second response time [20:11:00] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.516 second response time [20:11:11] RECOVERY - mail2 webmail.miraheze.org HTTPS on mail2 is OK: HTTP OK: HTTP/1.1 200 OK - 6162 bytes in 0.056 second response time [20:11:48] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:12:17] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:14:11] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.009 second response time [20:14:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [20:16:48] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.93, 8.33, 5.33 [20:16:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:20:41] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.07, 6.31, 5.26 [20:22:33] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 3.47, 5.65, 5.12 [20:22:39] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.83, 6.61, 5.51 [20:23:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.33, 2.82 [20:24:28] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 2.61, 4.62, 4.80 [20:24:38] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.62, 6.09, 5.44 [20:25:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.14, 3.30, 2.88 [20:31:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.57, 3.14 [20:33:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.48, 3.12, 3.02 [20:41:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.63, 3.27 [20:47:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.80, 3.17, 3.19 [20:49:21] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.12, 6.82, 6.27 [20:51:20] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.09, 6.61, 6.26 [20:53:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.84, 3.49 [20:55:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.72, 3.48 [20:57:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.09, 3.35, 3.37 [21:04:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.58, 3.48 [21:13:47] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.66, 3.99, 3.66 [21:14:41] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.18, 4.42, 2.45 [21:15:28] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [21:16:41] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 6.38, 5.50, 3.10 [21:17:28] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:18:07] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 15.01, 9.66, 5.64 [21:18:41] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.47, 4.68, 3.09 [21:19:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.68, 3.66 [21:21:04] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.068 second response time [21:21:11] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:21:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [21:21:26] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:21:26] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:21:41] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:18] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:19] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.020 second response time [21:22:41] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.08, 9.56, 5.52 [21:22:41] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:22:45] PROBLEM - mw8 APT on mw8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:46] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:23:00] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.17, 8.12, 6.97 [21:23:01] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23663 bytes in 2.385 second response time [21:23:08] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:23:08] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 2.086 second response time [21:23:22] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.355 second response time [21:23:36] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 0.602 second response time [21:24:40] RECOVERY - mw8 APT on mw8 is OK: APT OK: 19 packages available for upgrade (0 critical updates). [21:25:11] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:25:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.32, 3.15, 3.40 [21:25:40] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:26:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.63, 7.51, 7.01 [21:27:00] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:27:08] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 1.837 second response time [21:27:24] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:27:35] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.377 second response time [21:28:32] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.874 second response time [21:28:36] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.129 second response time [21:28:54] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 9.593 second response time [21:29:01] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.247 second response time [21:29:33] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.969 second response time [21:29:57] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.13, 6.89, 6.38 [21:30:41] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 2.83, 5.29, 5.16 [21:30:56] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.71, 6.26, 6.63 [21:31:54] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.56, 6.13, 6.17 [21:32:41] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 19.91, 9.33, 6.55 [21:33:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.99, 3.52, 3.51 [21:36:41] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 3.45, 5.95, 5.78 [21:37:36] dmehus: if you got questions, I'm not staying up late tonight [21:38:01] RhinosF1, ack, okay, sounds good [21:38:05] have a good sleep [21:38:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 2.51, 4.77, 6.00 [21:39:03] dmehus: my sleep will probably include a few nightmares. I'm sleeping so I can be ready for whatever craziness I'm definitely not expecting [21:39:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.80, 3.93, 3.63 [21:40:41] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 24.09, 10.98, 7.46 [21:41:43] RhinosF1, ah [21:43:07] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 6.02, 5.07, 3.94 [21:43:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.89, 3.70 [21:44:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 1.98, 3.05, 4.83 [21:44:15] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.37, 6.91, 6.08 [21:45:07] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 3.15, 4.28, 3.78 [21:45:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 4.04, 3.79 [21:46:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.64, 6.45, 6.02 [21:47:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.61, 3.67, 3.69 [21:50:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.02, 5.16, 5.13 [21:50:17] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [21:50:22] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:36] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:50:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:51:03] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:51:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.72, 2.76, 3.31 [21:51:12] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:52:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.36, 4.41, 4.85 [21:52:16] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:52:27] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23673 bytes in 9.505 second response time [21:52:34] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.200 second response time [21:52:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:53:05] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.037 second response time [21:53:10] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.666 second response time [21:58:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.51, 5.18, 5.01 [22:00:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.78, 4.61, 4.81 [22:01:56] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 23.75, 10.40, 6.09 [22:06:40] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:15:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:16:41] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 3.13, 4.64, 5.89 [22:19:07] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 2.70, 4.47, 5.73 [22:21:07] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 7.10, 5.05, 5.76 [22:22:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.47, 5.59, 4.54 [22:23:07] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 4.53, 4.80, 5.58 [22:24:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.64, 4.91, 4.42 [22:25:07] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 8.09, 5.24, 5.60 [22:26:41] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.04, 3.78, 4.88 [22:27:07] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 3.80, 4.47, 5.27 [22:29:08] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 3.01, 3.98, 4.99 [22:29:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.58, 3.25 [22:33:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.59, 3.37, 3.27 [22:43:35] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 4.76, 7.24, 5.88 [22:45:34] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 3.21, 5.85, 5.53 [22:46:07] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 4.42, 6.37, 5.27 [22:48:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.25, 5.71, 5.16 [22:50:07] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.04, 5.93, 5.30 [22:51:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.96, 3.43, 3.19 [22:51:29] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 5.85, 6.08, 5.65 [22:53:13] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 24.22, 14.08, 8.18 [22:54:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 3.92, 5.66, 5.39 [22:55:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.71, 4.53, 3.64 [22:55:27] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 3.07, 5.49, 5.63 [22:56:07] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 12.25, 8.48, 6.44 [22:59:24] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 2.28, 3.85, 4.92 [23:00:01] !log [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:02] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:03] !log [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost [23:00:04] !log [@mw11] starting deploy of {'l10nupdate': True} to ovlon [23:00:05] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [23:00:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 3.53, 5.85, 5.83 [23:00:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:03:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.29, 3.74, 3.67 [23:06:19] PROBLEM - test3 APT on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:07:55] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.23, 20.42, 16.93 [23:09:35] PROBLEM - test3 Current Load on test3 is CRITICAL: CRITICAL - load average: 8.63, 4.89, 2.14 [23:09:55] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.16, 20.95, 17.56 [23:12:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.48, 3.99, 4.90 [23:12:40] RECOVERY - test3 APT on test3 is OK: APT OK: 19 packages available for upgrade (0 critical updates). [23:13:12] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.98, 2.80, 3.26 [23:13:35] PROBLEM - test3 Current Load on test3 is WARNING: WARNING - load average: 1.09, 3.92, 2.49 [23:13:55] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 13.19, 17.98, 17.28 [23:15:35] RECOVERY - test3 Current Load on test3 is OK: OK - load average: 0.26, 2.67, 2.20 [23:16:07] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 5.91, 7.62, 6.35 [23:20:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 2.69, 5.06, 5.60 [23:20:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:22:07] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 10.12, 6.42, 5.97 [23:23:07] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 3.87, 4.22, 5.94 [23:23:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.28, 3.18 [23:24:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 3.88, 5.32, 5.63 [23:25:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.11, 3.46, 3.25 [23:25:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:27:07] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 6.37, 4.70, 5.68 [23:27:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.38, 3.40, 3.25 [23:29:07] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 3.46, 4.16, 5.36 [23:30:07] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 6.97, 5.17, 5.38 [23:32:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.43, 4.88, 5.25 [23:33:07] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 10.05, 6.79, 6.08 [23:33:25] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 2004s [23:33:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.69, 4.57, 5.10 [23:36:00] !log [@mw11] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 2156s [23:36:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:39:38] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-3 [23:39:39] [02dns] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbQXl [23:39:49] [02dns] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbQXl [23:39:51] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-2 [23:39:52] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbQXl [23:39:54] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [23:40:07] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 22.36, 9.94, 6.90 [23:45:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:52:42] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 4.14, 5.42, 4.69 [23:53:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.84, 3.39, 3.10 [23:53:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:54:41] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 14.46, 8.64, 5.91 [23:55:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.83, 3.17, 3.05 [23:58:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 3.23, 4.27, 5.91 [23:58:23] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.90, 1.62, 1.19 [23:58:41] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 2.87, 5.33, 5.15